Measuring RSS on Mac when MADV_FREE is used

[Jason Evans]

On Oct 11, 2011, at 7:57 AM, Justin Lebar wrote:
> At Mozilla, we recently ported our jemalloc fork to MacOS.  It works
> great, except we've noticed that our RSS measurements are now
> basically useless, because memory freed with madvise(MADV_FREE) is
> counted against our process's RSS until the system starts running low
> on memory and steals those pages away from us.
> 
> As far as I can tell, upstream jemalloc has the same behavior.
> 
> In contrast, MADV_DONTNEED on Linux releases memory (almost) immediately.
> 
> I don't think what MacOS is doing here is so unreasonable; the problem
> isn't so much that it's holding onto this memory, but rather that I
> now can't meaningfully measure my process's RSS via the operating
> system.  (The program can compute this value, of course, by measuring
> how much memory it's MADV_FREE'd.)
> 
> Do you guys know of any way I can force MacOS to drop my MADV_FREE'd
> pages, short of making a large allocation in another process (which
> might have other side-effects, like paging out things I want to keep
> in memory)?  Or is there a way to directly measure how many
> non-MADV_FREE'd pages my process is using?  I'm pretty stumped.

OS X's MADV_FREE behavior is great from a total system perspective.  jemalloc suffers nontrivial overhead on Linux because all MADV_DONTNEED'ed pages have to be zeroed before they are handed out again.  At Facebook we've considered numerous approaches to the problem, and tried a few of them, but I'd be thrilled to have MADV_DONTNEED on Linux.

I don't think there is a system-supplied way to introspect the number of pages with the MADV_FREE'd bit set.  If all you really care about is memory accounting, then note that starting with jemalloc 1.0.0, there are statistics counters that tell you roughly what you want to know.  In particular, take a look at the "stats.allocated", "stats.active", and "stats.mapped" statistics that are accessible via mallctl().  These and many other statistics are documented in the manual page:

	http://www.canonware.com/download/jemalloc/jemalloc-latest/doc/jemalloc.html

Internally, jemalloc knows precisely how many unused dirty pages could be MADV_FREE'd (see "stats.arenas.<i>.pdirty"), but once madvise(2) is called for a page, the application has no way of knowing whether the kernel actually reclaimed the page.  As such, you can monitor "stats.active" and have a pretty good idea of how much memory your application is using without counting the MADV_FREE'd pages.  You can also sum the "stats.arenas.<i>.pdirty" stats, but this isn't really necessary unless you want to be very precise, because unused dirty page accumulation is limited to be no more than 1/32 of active memory (by default).

Jason