Transparent Huge Pages

[Justin Lebar]

Hm, upon further consideration...

If you mmap a huge page (say, 1MB), then MADV_DONTNEED a few 4-KB
chunks inside, transparent huge pages should break up the huge page so
it can decommit the parts I asked it to decommit.  If it doesn't, that
sounds like a kernel bug to me!

Similarly, if I mmap 1MB, get a huge page, and then touch only a few
bytes in the middle, the kernel shouldn't commit a huge page.

If huge pages is behaving how I expect, I don't see why it would cause
your application to use more memory.

Just to check, you're measuring RSS, not vsize, right?

On Mon, Feb 20, 2012 at 4:59 PM, Justin Lebar <justin.lebar at gmail.com> wrote:
>> jemalloc seems to be prone to transparent huge pages
>> (https://lwn.net/Articles/423584), presumably due to its use of mmap().  In
>> my case (fuse module), the initial memory consumption jumped from ~12M to
>> ~27M.  The use of --enable-dss helps a little, bringing the consumption down
>> to ~19M.
>
> Ouch!
>
>> Did anyone else experienced similar behavior?  Is there an easy way of
>> avoiding transparent huge pages for jemalloc'ed memory?  The only workaround
>> that comes to my mind is a malloc wrapper that runs madvise(...,
>> MADV_NOHUGEPAGE) on every newly allocated chunk.
>
> You'd probably want to do this only on the 1MB chunks jemalloc
> allocates for small and tiny allocations.  For huge allocations (more
> than 1MB), it's likely the user will touch the whole thing, so huge
> pages could be a benefit.
>
>>
>> Cheers,
>> Jakob
>>
>> _______________________________________________
>> jemalloc-discuss mailing list
>> jemalloc-discuss at canonware.com
>> http://www.canonware.com/mailman/listinfo/jemalloc-discuss