jemalloc problem when working with dynamic library

[jokea]

What's the recommend way of using jemalloc with dynamic library?

I have a program which loads/unloads a liba.so many times using dlopen and dlclose.
The program starts with, say liba.so.1.0.0, and every time when I update the dynamic library,
say now it's liba.so.1.0.1, I send a message to the program, so it will call dlclose for liba.so.1.0.0 and
load the new liba.so.1.0.1 with dlopen. By doing this I get zero downtime while doing upgrade.

The code in liba.so contains many memory allocations and jemalloc(2.2.5) works better than tcmalloc and the
glibc's default malloc.

1. First I linked jemalloc against liba.so by compiling jemalloc using -fPIC flag.
However, after about 50 times of load/unload, dlopen returns an error:
"cannot allocate memory in static TLS block".
After some googling, I went to approach #2.

2. I compiled jemalloc using --disable-tls flag and the problem no longer exists(I'm not sure
disable tls will impact performance or not). However, this led to another problem,
after each load/unload, the memory usage of the program grows continuesly and eventually it aborted with error:
"<jemalloc>: Error in pthread_key_create()"
and malloc returns NULL when the program's RSS reaches about 90MB, but there are plenty of free memory on
the server when this happens.

3. Then I compiled jemalloc without -fPIC and --disable-tls flag and linked it against main, it works fine
this time. But in this way, I can not upgrade jemalloc if there's a newer version without terminating the main program.

So I'm wondering if #3 is the only right way to use jemalloc in such situation, and if I'm doing anything wrong
in #1 and #2.


Thanks,
Jokea