Iterating over DSS

[Jason Evans]

On 10/27/2011 02:15 AM, Dmitry Antipov wrote:
> is there a method to iterate over DSS heap and find all used/free regions?
> (I'm trying to see how much my DSS heap is fragmented). I.e. I would
> like to
> do something like:
>
> void *ptr;
> size_t size;
>
> malloc_mutex_lock(&dss_mtx);
> ptr = dss_base;
> fprintf(stderr, "DSS start: %p\n", ptr);
>
> while (ptr < dss_prev) {
> if (ptr_is_a_pointer_to_allocated_block) { /* How to check this ? */
> size = ivsalloc(ptr);
> fprintf(stderr, "used: %p..%p\n", ptr, ptr + size);
> } else
> ptr = (char *)ptr + sizeof (void *);
> }
>
> fprintf(stderr, "DSS end: %p\n", dss_prev);
> malloc_mutex_unlock(&dss_mtx);

jemalloc doesn't currently maintain the data structures necessary to 
iterate over all allocated objects.  If all runs within a chunk are 
completely full, then there is no way to find the chunk header, other 
than to start with a pointer to one of the allocated objects within the 
chunk.  Consider that jemalloc needs to be able to find unused space, 
but it never needs to introspect used space unless it already has some 
context to work in (freeing an existing object, expanding one, 
allocating an available region, etc.).  Several years ago jemalloc had 
the ability to iterate over allocated objects, but in practice it's 
rarely useful, and supporting it entails extra overhead, so I ripped the 
code out.

If you want to modify jemalloc to make iteration possible, I suggest 
enabling the rtree (radix tree) code and adding iterator functionality, 
so that you can find all the extant chunks.  Then you will need to write 
code that iterates over each chunk.  arena_chunk_purge() may give you 
some idea of how to iterate over chunks, though you will have to write 
custom code to iterate over small runs.

There are alternatives to consider.  The easiest is to use the 
statistics that jemalloc maintains if --enable-stats is specified at 
configuration time.  Somewhat more difficult is to create your own 
malloc wrapper and either log all allocation activity, or build up 
metadata that allow you to iterate over allocated objects.  I've done 
both of these things at various times (and even had log-based graphing 
tools based on jemalloc), but these approaches tend not to scale well 
for large or long-lived applications, so I stopped maintaining these 
tools once I had a good feel for the fragmentation characteristics of 
jemalloc's layout policies.

Jason