Jemalloc quickly OOMs with custom chunk_hooks that opt out of purge

[Jason Evans]

On Jun 15, 2016, at 10:17 AM, Benjamin Barg <benbakerbarg at gmail.com> wrote:
> I'm writing a set of chunk_hooks that satisfy chunk_alloc requests with a bump pointer and opt out of dalloc, decommit, and purge. The goal is to back an arena with 2MB and 1GB huge pages on demand and force jemalloc to reuse the memory in that space.
> 
> I wrote a simple program to test page reuse. As pseudocode:
> 	• malloc 1GB of memory using repeated small allocations
> 	• free that same 1GB
> 	• malloc a small amount of memory again
> 	• check whether jemalloc call chunk_alloc again in (3)
> I've noticed that when dirty page purging is disabled by setting MALLOC_CONF="lg_dirty_mult:-1", jemalloc willingly reuses all of the memory provided by my chunk hooks, but when the option is off, it only reuses about one chunk's worth (~4Mb).
> 
> I would consider this intended behavior (obviously my test program goes below the default 8:1 minimum active to free page ratio), except that the man page entry for chunk_purge_t reads
> 
> > A chunk purge function conforms to the chunk_purge_t type and optionally discards physical pages
> 
> I'm further confused because it seems that API for chunk_hooks_t offers no mechanism for signaling to jemalloc that the pages were not released.
> 
> Basically, I'm asking whether it's intended that opt.lg_dirty_mult must be set to -1 in order for jemalloc to reuse an arbitrary amount of chunk_hook-allocated memory.

I'm guessing that your chunk hook functions are opting out of purging, and that jemalloc is calling the deallocation hook.  If so, it's up to your chunk management code to deallocate (or reuse) that memory, so perhaps you're effectively leaking the memory in the chunk hook code.

Jason