network registered memory and pages_purge()

[D'Alessandro, Luke K]

On Jul 17, 2014, at 1:40 PM, Jason Evans <jasone at canonware.com> wrote:

> On Jul 17, 2014, at 10:29 AM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
>> We’re using jemalloc to manage network pinned memory using the custom chunk allocator/deallocator functionality. Essentially we decorate the default chunk allocator with memory registration calls. This mostly works great.
>> 
>> We just discovered though, that sometimes jemalloc will call pages_purge() on our chunks without telling us, which calls madvise().
>> 
>> [...]
>> 
>> This break our network registration, because the network card has already captured the virtual->physical mapping for the pages, and we’re not able to (nor do we want to) reregister mappings—it’s expensive and complex to keep track of.
>> 
>> We can mlock()/munlock() on some systems, but many of the systems we run on aren’t under our control and don’t like it when you mlock().
>> 
>> How important is this functionality? Can it be disabled at configure time? Would it make sense to add a dynamic configuration string to turn this off on a per-arena basis?
> 
> Assuming you're using jemalloc exclusively for managing the pinned memory, there's no problem at all with disabling the purging.  You can do the with MALLOC_CONF=lg_dirty_mult:-1 (or just change LG_DIRTY_MULT_DEFAULT in the source code, given that you already have other changes in place).

Thanks Jason, the environment variable works great. 

I don’t have any other changes in place, just using git master and the `arena.*.chunk.{alloc,dealloc}` functionality to register a function that forwards to the original {alloc,dalloc} and then uses IB verbs registration/deregistration on the returned chunks.

I don’t really want to to use jemalloc exclusively for managing the pinned memory—in fact, it gets used to back all of the normal malloc/free calls in the code. We use allocx() dallocx() with the pinned arena directly for network managed memory. Will the MALLOC_CONF cause us problems with the rest of the our runtime? I could link something else first, to deal with those routines if this is a bad thing to disable in general (-lc?).

> I'm planning to change dirty page purging quite a bit for jemalloc 4, in order to add more effective hysteresis.  The tuning mechanisms will change, but I expect it will still be possible to turn purging off completely.

Cool. It would be nice to have allocx() dallocx() take a “cache” reference instead of an arena reference, with the cache configured with arenas to handle cache misses or something to deal with multithreaded-accesses. Other than that we really like using the library and as long as our network memory doesn’t move between cores frequently, this works well.

Luke