RFC: TCMalloc-style new/delete hooks

[David Rigby]

Hi,

We are currently using TCMalloc as our memory allocator, however the significantly better fragmentation characteristics and deterministic lowest-available address selection of jemalloc means we want switching to jemalloc in the near future.

One of (the only?) sticking points however is the lack of a direct equivalent to TCMalloc’s new/delete hooks, which allow an application to register callbacks when memory is allocated/freed by the application. 

We use this feature to essentially perform sub-heap memory tracking, to determine how much memory different buckets (think tables/databases) are using. To be more specific, as a worker thread is assigned to a particular bucket the bucket ID is stored in TLS, and then when a new/delete callback is invoked we lookup the thread’s current bucket from TLS and increment/decrement the total used as appropriate.

To allow us to work with jemalloc, I’ve implemented[1] equivalent functionality in jemalloc.

I did consider making use of the arena functionality in jemalloc for this, but I was concerned about the potential increase in fragment ion with many arenas, which is exactly one of the reasons why we want to move away from TCMalloc (I’m proposing setting narenas=1 when we deploy).

How would you (Jason?) feel about merging this patch, or something conceptually similar into upstream? 


Thanks,

Dave Rigby


[1]: https://github.com/daverigby/jemalloc/commit/bbf3877d785417f03671bd1aed94723d750937d5