jemalloc on Solaris (Joyent SmartOS)

[Jason Evans]

On Sep 5, 2013, at 6:25 AM, Paul Smith <paul at mad-scientist.net> wrote:
> I have some software using jemalloc (in a large, heavily multi-threaded
> C++ program) currently running on Linux, MacOSX, and Windows, and we are
> porting it to Joyent's cloud-based SmartOS offering, which is Solaris
> underneath.  This is compiled using GCC 4.7.0 (that's the version
> packaged with this OS).
> 
> I'm using jemalloc 3.1.0-0-g3b1f3aca54fede23299cde9034f7b909c3d290d7
> currently; I'm not adverse to upgrading if that's warranted.
> 
> When I start my program it dumps core.  Examining the core dump the
> problem is obvious: my program is going into an infinite recursion
> trying to allocate memory.
> 
> Based on the stacktrace I rebuilt jemalloc without TLS support
> (--disable-tls), then I don't get this failure but I've not tested far
> enough to know if there might be other issues.

Any recursive call into malloc during early jemalloc bootstrapping will cause catastrophic failure.  This is unfortunately a common theme in jemalloc bootstrapping that requires OS-specific solutions.  OS X also calls malloc during TLS initialization, but TSD is very fast on OS X, so it's a reasonable alternative.  The fact that __emutls_get_address() (part of libgcc) is being called in this case makes me wonder if TLS is really an improvement over TSD.  In any case, the only expected adverse effect of disabling TLS is performance-related, and that is an OS-specific effect.  If you are concerned about a performance hit, you can use --with-jemalloc-prefix to change the allocation function names and write a microbenchmark that compares with/without TLS support built in.

> My main question is, I've noted that http://www.canonware.com/jemalloc/
> doesn't list Solaris as supported by the stand-alone jemalloc
> implementation, which is what I'm using.  A subsequent bullet item lists
> Solaris as being supported by the version of jemalloc that's included in
> Mozilla's FireFox allocator.
> 
> Is that still the situation today?  I'd really prefer to not try to use
> two different versions of jemalloc on my different platforms.  Is anyone
> familiar with the differences, and can they advise me on my best bet
> going forward?  Should I continue on working with stand-alone jemalloc
> on Solaris, or should I investigate using the version in FireFox?

The stand-alone version of jemalloc is far advanced beyond the version in Mozilla.

If you have general success with jemalloc on Solaris with --disable-tls and you determine that it's the right setting for all Solaris builds, please let me know so that we can force the option, as we do for OS X.  I don't have a Solaris environment set up, so I don't test releases there.  Your email is the first Solaris-related feedback I've received in quite some time, so up to now I've conservatively left it off the list of supported platforms.  There's already Solaris support in the configure script though, and I'm happy to take patches to get/keep jemalloc working on Solaris. 
 
Thanks,
Jason