RFC: TCMalloc-style new/delete hooks

[David Rigby]

- Dave

> On 14 Oct 2014, at 17:23, Daniel Micay <danielmicay at gmail.com> wrote:
> 
>> On 14/10/14 12:13 PM, David Rigby wrote:
>> Hi,
>> 
>> We are currently using TCMalloc as our memory allocator, however the significantly better fragmentation characteristics and deterministic lowest-available address selection of jemalloc means we want switching to jemalloc in the near future.
>> 
>> One of (the only?) sticking points however is the lack of a direct equivalent to TCMalloc’s new/delete hooks, which allow an application to register callbacks when memory is allocated/freed by the application. 
>> 
>> We use this feature to essentially perform sub-heap memory tracking, to determine how much memory different buckets (think tables/databases) are using. To be more specific, as a worker thread is assigned to a particular bucket the bucket ID is stored in TLS, and then when a new/delete callback is invoked we lookup the thread’s current bucket from TLS and increment/decrement the total used as appropriate.
>> 
>> To allow us to work with jemalloc, I’ve implemented[1] equivalent functionality in jemalloc.
>> 
>> I did consider making use of the arena functionality in jemalloc for this, but I was concerned about the potential increase in fragment ion with many arenas, which is exactly one of the reasons why we want to move away from TCMalloc (I’m proposing setting narenas=1 when we deploy).
>> 
>> How would you (Jason?) feel about merging this patch, or something conceptually similar into upstream? 
>> 
>> 
>> Thanks,
>> 
>> Dave Rigby
> 
> It seems like you could accomplish the same thing by setting a jemalloc
> prefix and then making wrappers without the prefix externally.
> 

So one added difficulty (which I should have mentioned) is Windows, where it doesn’t appear possible to interpose malloc et al like you can on most *ix systems.

Hence on Windows we will potentially stick to TCMalloc (which does a bunch of funky runtime-patching to interpose malloc), and would like to have a similar interface on with jemalloc (on *ix).

DaveR

> _______________________________________________
> jemalloc-discuss mailing list
> jemalloc-discuss at canonware.com
> http://www.canonware.com/mailman/listinfo/jemalloc-discuss