Measuring RSS on Mac when MADV_FREE is used

[Justin Lebar]

I'll keep hoping for a system-level command we can run, since doing
the adjustment in the browser makes it hard for our automated tools to
read Firefox's memory usage.

I recently modified our fork to always do MALLOC_DECOMMIT-style
accounting, so as of a few days ago, we have something like
stats.active.  :)  But since all of our allocations don't go through
jemalloc, we need to measure RSS, not just how big the heap is.

In the absence of any help from the OS, I may try to write a routine
which goes through and explicitly decommits and recommits MADV_FREE'd
pages.  (We're keeping track of these on mac now, since we always do
DECOMMIT-style accounting.)  If we call this before we record RSS, we
should get reasonable numbers.

An alternative would be to use mincore() to figure out whether the
MADV_FREE'd pages are in memory, and use that to adjust the reported
RSS.  This would probably be faster, but then we wouldn't be able to
measure RSS externally...

(For those following along at home, this work is happening in [1].)

In any case, thanks for your help, Jason!

I'm hoping we can move to upstream sometime soon, by the way.  You'll
hear from us in the form of patches if and when that happens.  :)

Regards,
-Justin

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=693404

On Tue, Oct 11, 2011 at 11:29 PM, Jason Evans <jasone at canonware.com> wrote:
> On Oct 11, 2011, at 7:57 AM, Justin Lebar wrote:
>> At Mozilla, we recently ported our jemalloc fork to MacOS.  It works
>> great, except we've noticed that our RSS measurements are now
>> basically useless, because memory freed with madvise(MADV_FREE) is
>> counted against our process's RSS until the system starts running low
>> on memory and steals those pages away from us.
>>
>> As far as I can tell, upstream jemalloc has the same behavior.
>>
>> In contrast, MADV_DONTNEED on Linux releases memory (almost) immediately.
>>
>> I don't think what MacOS is doing here is so unreasonable; the problem
>> isn't so much that it's holding onto this memory, but rather that I
>> now can't meaningfully measure my process's RSS via the operating
>> system.  (The program can compute this value, of course, by measuring
>> how much memory it's MADV_FREE'd.)
>>
>> Do you guys know of any way I can force MacOS to drop my MADV_FREE'd
>> pages, short of making a large allocation in another process (which
>> might have other side-effects, like paging out things I want to keep
>> in memory)?  Or is there a way to directly measure how many
>> non-MADV_FREE'd pages my process is using?  I'm pretty stumped.
>
> OS X's MADV_FREE behavior is great from a total system perspective.  jemalloc suffers nontrivial overhead on Linux because all MADV_DONTNEED'ed pages have to be zeroed before they are handed out again.  At Facebook we've considered numerous approaches to the problem, and tried a few of them, but I'd be thrilled to have MADV_DONTNEED on Linux.
>
> I don't think there is a system-supplied way to introspect the number of pages with the MADV_FREE'd bit set.  If all you really care about is memory accounting, then note that starting with jemalloc 1.0.0, there are statistics counters that tell you roughly what you want to know.  In particular, take a look at the "stats.allocated", "stats.active", and "stats.mapped" statistics that are accessible via mallctl().  These and many other statistics are documented in the manual page:
>
>        http://www.canonware.com/download/jemalloc/jemalloc-latest/doc/jemalloc.html
>
> Internally, jemalloc knows precisely how many unused dirty pages could be MADV_FREE'd (see "stats.arenas.<i>.pdirty"), but once madvise(2) is called for a page, the application has no way of knowing whether the kernel actually reclaimed the page.  As such, you can monitor "stats.active" and have a pretty good idea of how much memory your application is using without counting the MADV_FREE'd pages.  You can also sum the "stats.arenas.<i>.pdirty" stats, but this isn't really necessary unless you want to be very precise, because unused dirty page accumulation is limited to be no more than 1/32 of active memory (by default).
>
> Jason