Live allocation iterator

[Jason Evans]

Correct, jemalloc does not provide a mechanism for looking up interior pointers, nor does it have a mechanism for reliably determining whether a base pointer corresponds to a valid allocation.

> On Apr 20, 2015, at 3:45 AM, SNL <snl20465 at gmail.com> wrote:
> 
> Jason,
> 
> One more query on same line. 
> 
> Assume that base and extent are start and size of allocation. Is there infrastructure in Jemalloc to lookup an address (base + n ), n < extent ? My understanding is that such infrastructure does not exist in jemalloc. My use case involves checking whether a given pointer is part of a valid allocation or not. 
> 
> 
> 
> On Wed, Mar 25, 2015 at 2:14 PM, SNL <snl20465 at gmail.com> wrote:
> This looks relevant but never was pushed upstream.
> 
> https://www.mail-archive.com/jemalloc-discuss@canonware.com/msg00027.html
> 
> 
> Any thoughts on this patch set keeping in mind jemalloc-dev current state ? Any design/architectural inputs are also appreciated. This seems to have worked well for Alessandro who shared the patchset, I dont see these changes in their current code base though.
> 
> On Wed, Mar 25, 2015 at 11:25 AM, SNL <snl20465 at gmail.com> wrote:
> Does your use case allow you to wrap the allocator and keep a side table which supports iteration, or can you trace allocation activity and post-process it?
> 
> => This is what I have implemented currently but this does not scale for multithreaded programs due to locking issues. I was pondering over idea of not doing any book keeping myself and use jemalloc data structures ( and efficient low level locking ) for my purpose. 
> 
> Is this not possible even when all my metadata objects ( which hold user allocation base and extent ) are fixed sized and allocated from a separate arena ? I was under the impression that I should be able to mask and find out offsets within chunk->region->pages and find out objects. As my objects are all fixed size, they would be contiguous in memory. Is this valid understanding ? 
> 
> 
> On Tue, Mar 24, 2015 at 11:16 PM, Jason Evans <jasone at canonware.com> wrote:
> On Mar 24, 2015, at 6:53 AM, SNL <snl20465 at gmail.com> wrote:
>> If this is not possible ( I see an old post which says jemalloc does not have this infra but that was back in 2012 ). Is it possible to iterate over just an arena and dump all allocations in it  ? Basically, I have memory pool implemented on top of jemalloc which holds metadata about all other allocations jemalloc does, if I can iterate over this arena and get hold of metadata, I can get hold of each allocation as well. 
>> 
>> Any thoughts would be appreciated. 
>> 
>> On Tue, Mar 24, 2015 at 5:28 PM, SNL <snl20465 at gmail.com> wrote:
>> 
>> My use case is to walk through all live allocations ( the ones which are not freed yet) of all sizes the program ever made and dump it as the end of the program or on demand. I have looked at stats but I want something with even more details. Basically, the function should be able to iterate through all arenas and print details on following lines:
>> 
>> arena = 1 base = 0x7fd7beb88800 size = 15 bytes etc
>> 
>> Is there any api in jemalloc which already does subset of this  ? Any pointers will be helpful. 
> 
> jemalloc doesn't support iteration over live allocations, and it's unlikely to ever directly support iteration, because the feature imposes nontrivial additional complexity for limited benefit.  The closest thing under consideration is https://github.com/jemalloc/jemalloc/issues/203, but I don't plan to work on that until at least next year.
> 
> Does your use case allow you to wrap the allocator and keep a side table which supports iteration, or can you trace allocation activity and post-process it?
> 
> Jason
> 
> 
> 
> -- 
> 
> Cheers,
> Sunny. 
> 
> 
> 
> -- 
> 
> Cheers,
> Sunny. 
> 
> 
> 
> -- 
> 
> Cheers,
> Sunny.