Where do I modify jemalloc 3.5.0?

[Jason Evans]

On May 10, 2016, at 3:09 PM, sanjay khanna <khannasusa at gmail.com> wrote:
> I am using jemalloc 3.5.0 on my platform.  I have my own API layer on top of Jemalloc.
> On this platform (freebsd based), when a page is sbrk()'ed in, it is set to all zeros. This is given.
> 
> My API guarantees when Alloc is done by user code, bytes are all zero'ed out. So the user does not have to memset() or call calloc() for allocated memory. This is also a requirement on my API layer. If my alloc finds non-zero bytes in the allocated memory in the debug mode, it asserts because I presume that it is a write after free.
> 
> When a free() is done, my API layer memset(0) all the allocated bytes before calling jemalloc_free().
> This is also a security guarantee given by my API layer.
> 
> Right now at Alloc time, I am forced to call memset(0) as sometimes I find that there are unwanted non-zero bytes in the allocated memory. If my layer memset(0) before jemalloc_free() is called, why & where do these extra bytes appear from? These bytes are located in very high on a page address and look like jemalloc housekeeping.
> 
> Why do we do things like these?
>  Based on  my benchmark tests I know that it is taking me around 118ns (nanoseconds) to allocate a say 64 bytes using je_malloc() (without memset) and with a forced memset() this cost goes up to 240ns. That is almost double.and very expensive.
> 
> Since platform is already burning cpu to memset(0) on paging in a physical page first time, I think it is sub-optimal to  do additional memset(0) at alloc time.
> 
> If I were to modify/enhance jemalloc, where do I need to make changes? My list so far includes arena_run_split_large_helper(), arena_run_split_small() & chunk_recycle().

If I understand correctly, you want to rely on an invariant that "as long as all memory is zeroed prior to free(), subsequent malloc() calls will always return zeroed memory".  There are myriad ways this invariant is incompatible with jemalloc's internals.  Perhaps the most widely dispersed effect is that jemalloc calls e.g. imalloc() to allocate ephemeral metadata such as thread caches.  Another big issue is that runs used for small allocations directly embed metadata (no longer true in jemalloc 4).  Long story short, what you want to do isn't even close to being supported by vanilla jemalloc, and if you want to add such support you will need to audit pretty much the entirety of jemalloc to find ways it dirties memory, split APIs into internal/external versions, etc.

Jason