network registered memory and pages_purge()

[Jason Evans]

On Jul 17, 2014, at 10:29 AM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
> We’re using jemalloc to manage network pinned memory using the custom chunk allocator/deallocator functionality. Essentially we decorate the default chunk allocator with memory registration calls. This mostly works great.
> 
> We just discovered though, that sometimes jemalloc will call pages_purge() on our chunks without telling us, which calls madvise().
> 
> [...]
> 
> This break our network registration, because the network card has already captured the virtual->physical mapping for the pages, and we’re not able to (nor do we want to) reregister mappings—it’s expensive and complex to keep track of.
> 
> We can mlock()/munlock() on some systems, but many of the systems we run on aren’t under our control and don’t like it when you mlock().
> 
> How important is this functionality? Can it be disabled at configure time? Would it make sense to add a dynamic configuration string to turn this off on a per-arena basis?

Assuming you're using jemalloc exclusively for managing the pinned memory, there's no problem at all with disabling the purging.  You can do the with MALLOC_CONF=lg_dirty_mult:-1 (or just change LG_DIRTY_MULT_DEFAULT in the source code, given that you already have other changes in place).

I'm planning to change dirty page purging quite a bit for jemalloc 4, in order to add more effective hysteresis.  The tuning mechanisms will change, but I expect it will still be possible to turn purging off completely.

Jason