<div dir="ltr">Hi Alexey,<div> </div><div>I have run into similar issues (exorbitantly long wait times for the malloc mutex). The only way I could reduce the pain was to either reduce the number of threads, or reduce the amount of malloc/free() done by the application.</div> <div> </div><div>If you're using a library (like this xml one) it may be hard to reduce the number of malloc/frees() actually done.</div><div> </div><div>Can you try getting a full set of stack traces on the next stall (for all threads?). I have a feeling that your particular stack trace may be held up behind an arena_flush() but it'd be good to know for sure.</div> </div><div class="gmail_extra"> <div class="gmail_quote">On Tue, Jan 21, 2014 at 5:46 AM, Alexey Kuts <<a href="mailto:akuts@yandex-team.ru" target="_blank">akuts@yandex-team.ru</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, sometimes (maybe once per day) we have an annoying hang in jemalloc(). jemalloc-version: 3.4.1-0-g0135fb806e4137dc9cdf152541926a2bc95e33f0 Ubuntu 12.04.3 LTS, 3.2.0-33-generic x86_64 GNU/Linux #0 0x00007f6bd681a89c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f6bd6816080 in _L_lock_903 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f6bd6815f19 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x0000000000459127 in malloc_mutex_lock (mutex=0x7f6bd4400a08) at include/jemalloc/internal/mutex.h:77 #4 tcache_bin_flush_small (tbin=<optimized out>, binind=15, rem=3, tcache=0x7f6bd4006000) at src/tcache.c:105 #5 0x000000000045958d in tcache_event_hard (tcache=0x7f6bd4006000) at src/tcache.c:39 #6 0x0000000000437ebe in tcache_event (tcache=0x7f6bd4006000) at include/jemalloc/internal/tcache.h:271 #7 tcache_alloc_large (size=<optimized out>, tcache=<optimized out>, zero=<optimized out>) at include/jemalloc/internal/tcache.h:383 #8 arena_malloc (zero=false, size=<optimized out>, arena=0x0, try_tcache=true) at include/jemalloc/internal/arena.h:944 #9 imallocx (arena=0x0, try_tcache=true, size=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:798 #10 imalloc (size=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:807 #11 jemalloc (size=<optimized out>) at src/jemalloc.c:887 #12 0x00007f6bd6d3bad8 in xmlStrndup () from /usr/lib/x86_64-linux-gnu/libxml2.so.2 #13 0x00007f6bd6ce3029 in xmlNodeListGetString () from /usr/lib/x86_64-linux-gnu/libxml2.so.2 #14 0x00007f6bd6d0038f in xmlValidateElement () from /usr/lib/x86_64-linux-gnu/libxml2.so.2 #15 0x00007f6bd6d0034e in xmlValidateElement () from /usr/lib/x86_64-linux-gnu/libxml2.so.2 #16 0x00007f6bd6d0034e in xmlValidateElement () from /usr/lib/x86_64-linux-gnu/libxml2.so.2 #17 0x00007f6bd6d00528 in xmlValidateDtd () from /usr/lib/x86_64-linux-gnu/libxml2.so.2 #18 0x00007f6bd7005355 in xmlpp::DtdValidator::validate(xmlpp::Document const*) () from /usr/lib/libxml++-2.6.so.2 Will appreciate any hints or suggestions -- Kuts Alexey _______________________________________________ jemalloc-discuss mailing list <a href="mailto:jemalloc-discuss@canonware.com" target="_blank">jemalloc-discuss@canonware.com</a> <a href="http://www.canonware.com/mailman/listinfo/jemalloc-discuss" target="_blank">http://www.canonware.com/mailman/listinfo/jemalloc-discuss</a> </blockquote></div> </div>