Explicit huge pages

[Jason Evans]

On Sep 10, 2015, at 7:48 AM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
> I have an application that wants to use explicit huge pages during mmap. I have the infrastructure set up to do chunk allocation/deallocation from the hugetlbfs infrastructure, and we use it in custom arenas for network-registered memory allocation.
> 
> I’d like to also use these explicit huge pages for the default arenas. It’s fine with me if the vestigial chunks are allocated from 4k pages, so I can replace the chunk_hooks as necessary. My concern is related to jemalloc’s understanding of “page-size” as reported by arenas.page, dirty page purging, and --with-lg-page-sizes. Basically, I don’t understand enough about what is going on internal to jemalloc here.
> 
> If I end up using, e.g., 2GB pages and a 4GB chunk size, is there any point in enabling dirty-page purging? Would it even work? Do I need to tell jemalloc about the 2GB huge page size with the --with flag? The huge page size is a dynamic property—do I have to reconfigure jemalloc each time I want a different one?

I think the most promising approach is to leave jemalloc's notion of page size at 4 KiB, set the chunk size to be at least as large as the huge page size, and disable dirty page purging.  This allows the huge pages to be carved up with 4 KiB granularity for small/large allocations, and assures that chunks comprise distinct sets of huge pages.  Dirty page purging would be at best a waste of time in this set up, probably with no effect.

Jason