Rounding up huge allocations to page boundaries instead of chunks

[Jason Evans]

On Sep 9, 2014, at 6:51 AM, Guilherme Goncalves <ggp at mozilla.com> wrote:
> | Will this sufficiently address your accounting concerns?  There's the
> | potential to over-report active memory by nearly 1.2X in the worst case, but
> | that's a lot better than nearly 2X as things currently are.
> 
> While that's definitely better than 2X over-reporting, I wonder if we can't just expose the
> sum of all huge allocations rounded to a page boundary as a new statistic, without actually
> changing the way the mapping is done. That could give us the more accurate accounting we want
> without causing fragmentation in the address space.
> 
> In more concrete terms, this would add a "stats.arenas.<i>.huge.allocated_pages" statistic,
> reporting the total size of huge allocations serviced by the i-th arena, but rounded to pages
> and not chunks (while still mapping memory in chunks as usual).
> 
> If I'm not missing anything, a patch to implement this would look similar yet a lot less
> intrusive than my first attempt [1]. Does this sound reasonable?

I want the sum of malloc_usable_size() for all extant allocations to remain the source of truth about how much memory the application has allocated, and I'm currently on a mission to make size class spacing uniform, so I'm loath to add exceptions before even finishing that.  If 1.2X worst case is too loose a bound for your use case, one other possibility would be to add a configure option to create 8 size classes per size doubling rather than 4, so that the worst case is ~1.11X (or 16 size classes per doubling and 1.06X worst case overhead, etc.).  The size_classes.sh script requires only a single constant be parametrized in order to make this possible.

Thanks,
Jason