jemalloc initialization in a shared library

[Paul Smith]

On Thu, 2016-09-08 at 09:28 -0700, Jason Evans wrote:
> This may a separate issue from the TLS initialization issue you're
> hitting, but linking a malloc implementation into a dlopen()ed
> library is exceedingly difficult to make work correctly, because it's
> very difficult to avoid mixed allocator use, e.g. calling malloc() of
> one implementation and erroneously calling free() of the other.

As a followup, I spent a few days working with the user including
providing them with a version of the library built with no
optimization.  When the hang happens with that version the stacktrace
is very different and appears to be happening trying to obtain the C++
STL's TLS variable that manages exceptions etc. (__cxa_get_global etc.)
and jemalloc is not implicated any longer.

The mutex that is deadlocked appears to be the one taken by the runtime
linker code that loads shared objects.  It's really not clear to me at
all why this code is even being invoked at this point, since we're
definitely not loading objects.

However, armed with this information I suggested to the user that they
force our library to be loaded at startup time of the JVM by using
LD_PRELOAD, and that seems to have solved their problem.

Until I get some way to reproduce the problem myself I don't think it
will be productive for me to work on this further.  However, during
this time I did manage to update our version of jemalloc to 4.2.1,
although it still needs more testing before deploying, so it wasn't all
for nothing! :).

Thanks for the thoughtful suggestions, I do appreciate it.