hang in jemalloc()

[Alexey Kuts]

our case is always single threaded (it-is mapreduce-job).

22.01.2014 21:30, John Esmet пишет:
> Hi Alexey,
>
> I have run into similar issues (exorbitantly long wait times for the
> malloc mutex). The only way I could reduce the pain was to either reduce
> the number of threads, or reduce the amount of malloc/free() done by the
> application.
>
> If you're using a library (like this xml one) it may be hard to reduce
> the number of malloc/frees() actually done.
>
> Can you try getting a full set of stack traces on the next stall (for
> all threads?). I have a feeling that your particular stack trace may be
> held up behind an arena_flush() but it'd be good to know for sure.
>
>
> On Tue, Jan 21, 2014 at 5:46 AM, Alexey Kuts <akuts at yandex-team.ru
> <mailto:akuts at yandex-team.ru>> wrote:
>
>     Hi,
>
>     sometimes (maybe once per day) we have
>     an annoying hang in jemalloc().
>
>     jemalloc-version: 3.4.1-0-__g0135fb806e4137dc9cdf152541926__a2bc95e33f0
>     Ubuntu 12.04.3 LTS, 3.2.0-33-generic x86_64 GNU/Linux
>
>
>     #0  0x00007f6bd681a89c in __lll_lock_wait () from
>     /lib/x86_64-linux-gnu/__libpthread.so.0
>     #1  0x00007f6bd6816080 in _L_lock_903 () from
>     /lib/x86_64-linux-gnu/__libpthread.so.0
>     #2  0x00007f6bd6815f19 in pthread_mutex_lock () from
>     /lib/x86_64-linux-gnu/__libpthread.so.0
>     #3  0x0000000000459127 in malloc_mutex_lock (mutex=0x7f6bd4400a08)
>     at include/jemalloc/internal/__mutex.h:77
>     #4  tcache_bin_flush_small (tbin=<optimized out>, binind=15, rem=3,
>     tcache=0x7f6bd4006000) at src/tcache.c:105
>     #5  0x000000000045958d in tcache_event_hard (tcache=0x7f6bd4006000)
>     at src/tcache.c:39
>     #6  0x0000000000437ebe in tcache_event (tcache=0x7f6bd4006000) at
>     include/jemalloc/internal/__tcache.h:271
>     #7  tcache_alloc_large (size=<optimized out>, tcache=<optimized
>     out>, zero=<optimized out>) at include/jemalloc/internal/__tcache.h:383
>     #8  arena_malloc (zero=false, size=<optimized out>, arena=0x0,
>     try_tcache=true) at include/jemalloc/internal/__arena.h:944
>     #9  imallocx (arena=0x0, try_tcache=true, size=<optimized out>) at
>     include/jemalloc/internal/__jemalloc_internal.h:798
>     #10 imalloc (size=<optimized out>) at
>     include/jemalloc/internal/__jemalloc_internal.h:807
>     #11 jemalloc (size=<optimized out>) at src/jemalloc.c:887
>     #12 0x00007f6bd6d3bad8 in xmlStrndup () from
>     /usr/lib/x86_64-linux-gnu/__libxml2.so.2
>     #13 0x00007f6bd6ce3029 in xmlNodeListGetString () from
>     /usr/lib/x86_64-linux-gnu/__libxml2.so.2
>     #14 0x00007f6bd6d0038f in xmlValidateElement () from
>     /usr/lib/x86_64-linux-gnu/__libxml2.so.2
>     #15 0x00007f6bd6d0034e in xmlValidateElement () from
>     /usr/lib/x86_64-linux-gnu/__libxml2.so.2
>     #16 0x00007f6bd6d0034e in xmlValidateElement () from
>     /usr/lib/x86_64-linux-gnu/__libxml2.so.2
>     #17 0x00007f6bd6d00528 in xmlValidateDtd () from
>     /usr/lib/x86_64-linux-gnu/__libxml2.so.2
>     #18 0x00007f6bd7005355 in
>     xmlpp::DtdValidator::validate(__xmlpp::Document const*) () from
>     /usr/lib/libxml++-2.6.so.2
>
>     Will appreciate any hints or suggestions
>
>     --
>     Kuts Alexey
>     _________________________________________________
>     jemalloc-discuss mailing list
>     jemalloc-discuss at canonware.com <mailto:jemalloc-discuss at canonware.com>
>     http://www.canonware.com/__mailman/listinfo/jemalloc-__discuss
>     <http://www.canonware.com/mailman/listinfo/jemalloc-discuss>
>
>

-- 
Куц Алексей
http://staff.yandex-team.ru/akuts