Deadlock during application startup (jemalloc with prof:true)

[Jason Evans]

On Jan 20, 2014, at 1:03 AM, Evgeniy Ivanov <i at eivanov.com> wrote:
> I got a following deadlock:
> 
> #0  __lll_lock_wait_private () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
> #1  0x00007f6f72cc048f in _L_lock_16 () from
> /home/eiva/prefix/glibc-2.16/lib/libc.so.6
> #2  0x00007f6f72cc02d1 in __new_exitfn (listp=0x7f6f73030c80 <lock>)
> at cxa_atexit.c:78
> #3  0x00007f6f72cc0424 in __internal_atexit (func=0x7f6f7371aad0
> <prof_fdump>, arg=0x0, d=0x7f6f73932570, listp=<optimized out>) at
> cxa_atexit.c:34
> #4  0x00007f6f7371a9d7 in prof_boot2 () at
> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/prof.c:1212
> #5  0x00007f6f736f3c4e in malloc_init_hard () at
> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/jemalloc.c:790
> #6  0x00007f6f736effc4 in malloc_init () at
> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/jemalloc.c:306
> #7  0x00007f6f736f0698 in calloc (num=1, size=1040) at
> /export/home/eiva/git/thd/src/thirdparty/jemalloc/3.4.1/src/src/jemalloc.c:1036
> #8  0x00007f6f72cc03ba in __new_exitfn (listp=0x7f6f7302f5a8
> <__exit_funcs>) at cxa_atexit.c:100
> #9  0x00007f6f72cc0424 in __internal_atexit (func=0x3972062ae0
> <std::ios_base::Init::~Init()>, arg=0x7f6f736ea9c8 <_ZStL8__ioinit>,
> d=0x7f6f736df170, listp=<optimized out>) at cxa_atexit.c:34
> #10 0x00007f6f7342336e in __cxx_global_var_init () at
> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/iostream:72
> #11 0x00007f6f73423389 in global constructors keyed to a() () at
> /tb/builds/thd/sbn/2.6/src/thirdparty/quickfix/1.12.4/src/src/C++/SocketInitiator.cpp:54
> #12 0x00007f6f7348aa36 in __do_global_ctors_aux () from
> //home/eiva/git/tb/build.x86_64-unknown-linux/platform/lib64/libquickfix.so.10
> #13 0x00007f6f733cdb43 in _init () from
> //home/eiva/git/tb/build.x86_64-unknown-linux/platform/lib64/libquickfix.so.10
> #14 0x00007ffff9915058 in ?? ()
> #15 0x00007f6f73953849 in call_init (l=0x7f6f73b65540,
> argc=1936584824, argv=0x7ffff9915048, env=0x7ffff9915058) at
> dl-init.c:69
> #16 0x00007f6f73953987 in _dl_init (main_map=0x7f6f73b691c8, argc=1,
> argv=0x7ffff9915048, env=0x7ffff9915058) at dl-init.c:133
> #17 0x00007f6f73945b2a in _dl_start_user () from
> /home/eiva/prefix/glibc-2.16/lib/ld-2.16.so
> #18 0x0000000000000001 in ?? ()
> #19 0x00007ffff9917072 in ?? ()
> #20 0x0000000000000000 in ?? ()
> 
> It is always reproducible with libquickfix-1.12.4 linked against my
> application (libc-2.12 and libc-2.16).
> MALLOC_CONF="prof:true"
> 
> Is it issue in jemalloc or in glibc?

libquickfix is registering an atexit() function inside a library initializer, prior to any other allocation having triggered jemalloc initialization.  During jemalloc initialization, there are two potential atexit() calls, one for heap profiling and one for printing stats at exit.  The deadlock here is occurring due to glibc's atexit() code holding a lock during the calloc() call that triggers jemalloc initialization, and jemalloc trying to call back into the atexit() code.

There's little jemalloc can do about this except to avoid atexit().  However, you may still be able to use heap profiling in conjunction with libquickfix.  I vaguely recall that it is possible to specify precedences for library initializers; you can probably free(malloc(1)) inside a high priority library initializer inside your own application code and cause jemalloc to be initialized prior to libquickfix, in which case the recursive atexit() deadlock will be avoided.

Jason