jemalloc hooks clarifications

[Jason Evans]

On Jan 19, 2016, at 9:56 AM, Jakob Buchgraber <bucjac at gmail.com> wrote:
> I think there might be an issue with this approach though: https://github.com/jemalloc/jemalloc/issues/307

I'll try not to sit on this issue indefinitely, but it takes a while to get my head wrapped around the chunk hooks, so it may be a while before I have time to dig into it.  So much to do...

> It’s deadlocking right now as well, as I am accessing stats from within the chunk hooks to determine which arenas to purge. I had to replace the malloc mutexes with recursive mutexes to make it work. Seems fine so far.

My memory is now fuzzy on this aspect of chunk hooks, but my vague recollection is that I attempted to drop all locks before calling into application-provided hook functions, mainly to avoid resource starvation in case the hook functions block in system calls.  Maybe that's not generally possible, or maybe there's just a bug somewhere.  In any case, if allocation is triggered by a call into mallctl(), calling back into mallctl() inside the hooks is going to deadlock, so calling back into jemalloc APIs is not generally safe.

> Basically, I am running with lots of main memory (> 1TB). Most of the time the program will only use a fraction of the available memory but some queries will require almost all the memory in some random arena. So even if I leave purging on and set lg_dirty_mult to say 3, some arenas might end up having cached 10s of GB of physical memory with others running out and the program will crash. Ideally, I would want BSD’s MADV_FREE on Linux. That patch never got merged though. So what I am doing is to add some logic that tracks the amount of committed physical memory and if some threshold is reached, I query the jemalloc stats and dynamically adjust the purging ratio. Does that make sense?

Yes, dynamically updating the purging ratio is a reasonable approach, and I've done something similar in HHVM (temporarily purge more aggressively if any request threads go idle).

By the way, the MADV_FREE will apparently be in the next Linux kernel release.

Keep an eye on https://github.com/jemalloc/jemalloc/issues/149, which I'm actively working on.  So far it looks like I'm going to be able to implement a time-based purging mechanism that does a much better job of controlling unused dirty pages *without* asynchronous purging threads, as long as the application doesn't go completely idle.  I'm planning to get this feature into the 4.1.0 release as an opt-in, and I suspect it's going to prevail as the default mechanism in a subsequent release.

Jason