More aggressive page purging

[Jason Evans]

On May 10, 2012, at 11:55 AM, Justin Lebar wrote:
>>> In summary, I don't think there's a problem here to fix.  Am I missing
>>> something?
>> 
>> We could use RSS - number of madvised pages, for sure, but that doesn't
>> quite help with people looking at their task manager ans seeing memory
>> usage 1GB higher than what it actually is. (And it doesn't help to make
>> the numbers Firefox itself reports believable)
> 
> Yeah, the perception problem ("I looked at the task manager and
> Firefox sux!") is a real one that, from MemShrink's perspective, we
> need to solve.

It's unfortunate that the operating systems don't expose statistics that provide clarity here.  I don't want to make aggressive purging default behavior, but I'm open to patches that make this a configuration-time behavior after the 3.0.0 release (probably today or tomorrow).

Re: stealing a bit for 'decommit', overloading the 'large' bit is probably the easiest approach.  The recent arena_mapbits_*() refactoring should make decommit support much easier to add.

> But also, |RSS minus madvised memory| is wrong as soon as any of the
> madvised memory is kicked out due to memory pressure.  So without
> double-purge semantics, I don't see how you'd be able to reliably
> measure "hard" memory usage.

jemalloc maintains 'allocated', 'active', and 'mapped' statistics.  'active' ignores a bit of internally allocated metadata memory, but it is very close to being RSS minus madvise'd memory minus non-malloc memory (libraries, other mapped files, etc.).

Jason