More aggressive page purging

[Jason Evans]

On May 10, 2012, at 6:50 AM, Mike Hommey wrote:
> On Thu, May 10, 2012 at 12:00:18PM +0200, Mike Hommey wrote:
>> On Thu, May 10, 2012 at 11:51:24AM +0200, Mike Hommey wrote:
>>> Hi,
>>> 
>>> madvise(MADV_FREE), on OSX, is not quite like madvise(MADV_DONTNEED) on
>>> Linux: it doesn't actually do anything until the kernel really needs to
>>> because of some memory pressure (or at least it looks like so). This
>>> means that even when triggering an arenas.purge with mallctl, we can't
>>> rely on the RSS value we get.
>>> 
>>> One way to make pages purging more aggressive is to use mmap(MAP_FIXED)
>>> instead of madvise. As far as I can tell, it doesn't have an effect on
>>> performance, at least not on Firefox.
>>> 
>>> Do you think we could reasonably switch pages_purge to use
>>> mmap(MAP_FIXED) on OSX, or would you prefer a configure option?
>> 
>> In fact, it seems I talked too soon. it seems there *is* a performance
>> impact. However, I think having mallctl("arenas.purge") trigger
>> mmap(MAP_FIXED) instead of madvise() on OSX would be helpful, and
>> wouldn't hurt.
> 
> As a matter of fact, pages_purge also doesn't decrease RSS on Windows.
> We do have two different ways to handle the situation in Firefox with
> the old jemalloc: decommit (for windows) and double purge (for OSX).
> 
> In both cases, a flag is added to the flags stored with a run address.
> 
> - Decommit: when a page is purged, it is decommitted (which means
>  the address space is still reserved, but accesses will trigger an
>  exception), and flagged as such. When allocating from such decommitted
>  pages, jemalloc needs to commit them beforehand.
> 
> - Double purge: when a page is purged, madvise(MADV_FREE) is used, which
>  means the OS is free to drop the page in memory pressure situations.
>  The page is flagged as decommitted. When allocating from that page, it
>  is unmarked as decommitted. Nothing else needs to happen on the page
>  (desides maybe emptying it if we zeroing is enabled). There is an
>  additional function that triggers a full purge, in which case we scan
>  the arenas for the decommitted pages, and mmap(MAP_FIXED) over them.
> 
> I think we should have one of the above in jemalloc3 to handle both
> platforms. As there is really nothing to "lightly" decommit on OSX, we
> should go with the latter, with the function triggerring the "decommit"
> being mallctl("arenas.purge").
> 
> Since this requires an additional flag, we basically have two choices:
> reuse the "large" flag, which is unused in unallocated runs. Or we can
> use any one of the bits [4-11].
> 
> What do you think?


I don't see the point in double purge on OS X (or FreeBSD, which behaves similarly), assuming that the kernel actually does the right think under memory pressure.  All it does is make 'top' output easier to interpret,  yet jemalloc provides adequate statistics to accurately assess the application's actual active memory footprint (and Firefox even exposes those statistics, right?).  IMO MADV_FREE is a feature, not a bug.  Indeed, on Linux, MADV_DONTNEED is a performance headache that we've had to work around in various ways at Facebook, with mixed success.

My memory of Windows virtual memory semantics is fuzzy, but my recollection is that the decommit support we put into jemalloc for Firefox was motivated primarily by inadequate operating system statistics for Windows XP that confused our understanding of actual memory usage.  The VirtualAlloc() documentation indicates that MEM_RESET has the same semantics as MADV_FREE, which is (again IMO) a good thing.

In summary, I don't think there's a problem here to fix.  Am I missing something?

Thanks,
Jason