hang in jemalloc()

[Jason Evans]

On Jan 21, 2014, at 2:46 AM, Alexey Kuts <akuts at yandex-team.ru> wrote:
> sometimes (maybe once per day) we have
> an annoying hang in jemalloc().
> 
> jemalloc-version: 3.4.1-0-g0135fb806e4137dc9cdf152541926a2bc95e33f0
> Ubuntu 12.04.3 LTS, 3.2.0-33-generic x86_64 GNU/Linux
> 
> 
> #0  0x00007f6bd681a89c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007f6bd6816080 in _L_lock_903 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #2  0x00007f6bd6815f19 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
> #3  0x0000000000459127 in malloc_mutex_lock (mutex=0x7f6bd4400a08) at include/jemalloc/internal/mutex.h:77
> #4  tcache_bin_flush_small (tbin=<optimized out>, binind=15, rem=3, tcache=0x7f6bd4006000) at src/tcache.c:105
> #5  0x000000000045958d in tcache_event_hard (tcache=0x7f6bd4006000) at src/tcache.c:39
> #6  0x0000000000437ebe in tcache_event (tcache=0x7f6bd4006000) at include/jemalloc/internal/tcache.h:271
> #7  tcache_alloc_large (size=<optimized out>, tcache=<optimized out>, zero=<optimized out>) at include/jemalloc/internal/tcache.h:383
> #8  arena_malloc (zero=false, size=<optimized out>, arena=0x0, try_tcache=true) at include/jemalloc/internal/arena.h:944
> #9  imallocx (arena=0x0, try_tcache=true, size=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:798
> #10 imalloc (size=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:807
> #11 jemalloc (size=<optimized out>) at src/jemalloc.c:887
> #12 0x00007f6bd6d3bad8 in xmlStrndup () from /usr/lib/x86_64-linux-gnu/libxml2.so.2
> #13 0x00007f6bd6ce3029 in xmlNodeListGetString () from /usr/lib/x86_64-linux-gnu/libxml2.so.2
> #14 0x00007f6bd6d0038f in xmlValidateElement () from /usr/lib/x86_64-linux-gnu/libxml2.so.2
> #15 0x00007f6bd6d0034e in xmlValidateElement () from /usr/lib/x86_64-linux-gnu/libxml2.so.2
> #16 0x00007f6bd6d0034e in xmlValidateElement () from /usr/lib/x86_64-linux-gnu/libxml2.so.2
> #17 0x00007f6bd6d00528 in xmlValidateDtd () from /usr/lib/x86_64-linux-gnu/libxml2.so.2
> #18 0x00007f6bd7005355 in xmlpp::DtdValidator::validate(xmlpp::Document const*) () from /usr/lib/libxml++-2.6.so.2
> 
> Will appreciate any hints or suggestions

I just audited the mutex operations, and I see no way that a bin mutex can enter the locked state without later being unlocked.  Is your application using the allocator inside signal handlers, by any chance?

Jason