Using jemalloc with static link

[Jason Evans]

On Jul 19, 2013, at 12:41 AM, Colin Hercus <colin at novocraft.com> wrote:
> I am trying to test a HPC appl'n with jemalloc to see if it can reduce memory fragmentation and slow memory growth of a long running multi-threaded app.
> 
> This application is statically linked as it's not open source and we need one binary distribution to run on multiple Linux versions.
> 
> If I configure jemalloc without a jemalloc prefix (je_) and link with -static -ljemalloc I get linker errors for multiply defined symbols
> 
> g++  -m64 -Wl,--eh-frame-hdr -o ./bin/xxxx ./obj/xxxx.o  .....   -static -pthread -lcrypto -lz -lbz2 -ldl  -ljemalloc -lunwind -Wl,-Map=gccmaps/xxxx.map

There's a good chance that you can make the errors go away by putting -ljemalloc earlier in the link line.  It looks to me like libcrypto is pulling in the glibc malloc symbols, and by the time the libjemalloc portion of linking occurs, it's too late for jemalloc to provide the symbols.  That said, pure static linking is pretty perilous, and I recommend dynamically linking against the system libraries, and linking to static libraries by specifying e.g. libjemalloc_pic.a along with the .o object files (and/or .a archives) in your project.  glibc in particular does an excellent job of maintaining symbol compatibility across many years of releases, so you can ship around a binary that is dynamically linked against glibc with fewer issues than you will run into with a statically linked-in glibc.  glibc protects you from a rather unstable Linux kernel interface, and if you try to run your static binary on top of a substantially different kernel, hilarity will ensue.

> If I build with a prefix of je_ then linking is okay but memory allocation is via glibc malloc. The same is true if I dynamically link without a prefix.
> 
> Is there any simple easy way to get je_malloc to hook itself in as a replacement for malloc the way tcmalloc does?

jemalloc *does* hook itself in as a replacement for glibc malloc, but only if no prefix is specified. =)  You can make a one line change to src/jemalloc.c to unconditionally enable hooking.  Replace:

#if ((is_malloc(je_malloc) == 1) && defined(__GLIBC__) && !defined(__UCLIBC__))

with:

#if 1

Jason