More aggressive page purging

[Mike Hommey]

On Thu, May 10, 2012 at 12:00:18PM +0200, Mike Hommey wrote:
> On Thu, May 10, 2012 at 11:51:24AM +0200, Mike Hommey wrote:
> > Hi,
> > 
> > madvise(MADV_FREE), on OSX, is not quite like madvise(MADV_DONTNEED) on
> > Linux: it doesn't actually do anything until the kernel really needs to
> > because of some memory pressure (or at least it looks like so). This
> > means that even when triggering an arenas.purge with mallctl, we can't
> > rely on the RSS value we get.
> > 
> > One way to make pages purging more aggressive is to use mmap(MAP_FIXED)
> > instead of madvise. As far as I can tell, it doesn't have an effect on
> > performance, at least not on Firefox.
> > 
> > Do you think we could reasonably switch pages_purge to use
> > mmap(MAP_FIXED) on OSX, or would you prefer a configure option?
> 
> In fact, it seems I talked too soon. it seems there *is* a performance
> impact. However, I think having mallctl("arenas.purge") trigger
> mmap(MAP_FIXED) instead of madvise() on OSX would be helpful, and
> wouldn't hurt.

As a matter of fact, pages_purge also doesn't decrease RSS on Windows.
We do have two different ways to handle the situation in Firefox with
the old jemalloc: decommit (for windows) and double purge (for OSX).

In both cases, a flag is added to the flags stored with a run address.

- Decommit: when a page is purged, it is decommitted (which means
  the address space is still reserved, but accesses will trigger an
  exception), and flagged as such. When allocating from such decommitted
  pages, jemalloc needs to commit them beforehand.

- Double purge: when a page is purged, madvise(MADV_FREE) is used, which
  means the OS is free to drop the page in memory pressure situations.
  The page is flagged as decommitted. When allocating from that page, it
  is unmarked as decommitted. Nothing else needs to happen on the page
  (desides maybe emptying it if we zeroing is enabled). There is an
  additional function that triggers a full purge, in which case we scan
  the arenas for the decommitted pages, and mmap(MAP_FIXED) over them.

I think we should have one of the above in jemalloc3 to handle both
platforms. As there is really nothing to "lightly" decommit on OSX, we
should go with the latter, with the function triggerring the "decommit"
being mallctl("arenas.purge").

Since this requires an additional flag, we basically have two choices:
reuse the "large" flag, which is unused in unallocated runs. Or we can
use any one of the bits [4-11].

What do you think?

Mike