Patch: enumerate all live blocks

[Benoit Jacob]

It is equivalent, as long as you can ensure that _all_ allocations go
through your wrappered malloc. The terror of anyone making such a
wrappered malloc is that some allocations might escape it, such as:
allocations made by libraries that your application uses, etc. Not
being familiar with the field, I just found it easier to be confident
that all allocations would be correctly wrappered by doing it in the
memory allocator itself, since if any allocation did not go through
it, we would have allocator mismatches anyway.

Here is an example of what scared me away from trying to do it in the
application: this is a comment in jemalloc 3.0's src/jemalloc.c:

#if ((is_malloc(je_malloc) == 1) && defined(__GLIBC__) && !defined(__UCLIBC__))
/*
 * glibc provides the RTLD_DEEPBIND flag for dlopen which can make it possible
 * to inconsistently reference libc's malloc(3)-compatible functions
 * (https://bugzilla.mozilla.org/show_bug.cgi?id=493541).
 *
 * These definitions interpose hooks in glibc.  The functions are actually
 * passed an extra argument for the caller return address, which will be
 * ignored.
 */
JEMALLOC_EXPORT void (* const __free_hook)(void *ptr) = je_free;
JEMALLOC_EXPORT void *(* const __malloc_hook)(size_t size) = je_malloc;
JEMALLOC_EXPORT void *(* const __realloc_hook)(void *ptr, size_t size) =
    je_realloc;
JEMALLOC_EXPORT void *(* const __memalign_hook)(size_t alignment, size_t size) =
    je_memalign;
#endif

Benoit

2012/10/6 Salvatore Sanfilippo <antirez at gmail.com>:
> Hello,
>
> what is the advantage of this approach compared to doing it entirely
> in the application code just wrapping malloc/realloc/free?
> Basically wrappered_malloc() allocates a bit more space accordingly to
> the metadata to store, store this metadata at the start of the block,
> and then returns the pointer advanced to the start of the empty space.
>
> Doing it in the context of the application makes it
> malloc-implementation agnostic that can be an advantage.
>
> Regards,
> Salvatore
>
> On Sat, Oct 6, 2012 at 7:28 AM, Benoit Jacob <jacob.benoit.1 at gmail.com> wrote:
>> Hello,
>>
>> The attached patch instruments jemalloc 3.0, adding the ability to
>> enumerate all live blocks.
>>
>> Currently, the only information stored about blocks is their (payload)
>> address and size, but the plan is to also store their allocation call
>> stack.
>>
>> This is achieved by allocating larger blocks than requested, and using
>> the extra space to store doubly linked list elements.
>>
>>  This is provided just in case it might be useful to anyone, not
>> considered ready for inclusion in jemalloc. It's been tested for 15
>> minutes in a Firefox build.
>>
>> Details about the overhead, and how it could be reduced:
>>  * Memory overhead is, per block: 32 bytes on 32-bit systems, 48 bytes
>> on 64-bit systems (assuming size_t == uintptr_t). Could easily be
>> reduced to 16 bytes in both cases (by using a XOR linked list and
>> assuming that no block exceeds 4G).
>>  * Slowness overhead is essentially an additional mutex to lock on
>> every malloc/free call. Could be solved in various ways, either
>> copying what jemalloc does internally, or by using a lock-free list.
>>
>> If you want to test it out, currently the only built-in way to output
>> the list of blocks is to call je_dump_list_of_blocks(void) e.g. from
>> your debugger. See its code to see the relevant calls. Sample output
>> from Firefox:
>>
>> ...snip...
>> Block 193965:  real block = 0x7fffba218580,  payload = 0x7fffba2185b0,
>>  payload size = 64
>> Block 193966:  real block = 0x7fffc02dd000,  payload = 0x7fffc02dd030,
>>  payload size = 1024
>> Block 193967:  real block = 0x7fffc2053ce0,  payload = 0x7fffc2053d10,
>>  payload size = 24
>> Block 193968:  real block = 0x7fffc5b4ed80,  payload = 0x7fffc5b4edb0,
>>  payload size = 80
>> Block 193969:  real block = 0x7fffd119e240,  payload = 0x7fffd119e270,
>>  payload size = 64
>> Block 193970:  real block = 0x7fffcfb4aa60,  payload = 0x7fffcfb4aa90,
>>  payload size = 24
>> Block 193971:  real block = 0x7fffbb85e1f0,  payload = 0x7fffbb85e220,
>>  payload size = 24
>> Block 193972:  real block = 0x7fffe20a62e0,  payload = 0x7fffe20a6310,
>>  payload size = 32
>>
>> End enumeration of 193973 jemalloc blocks.
>>
>> Cheers,
>> Benoit
>>
>> _______________________________________________
>> jemalloc-discuss mailing list
>> jemalloc-discuss at canonware.com
>> http://www.canonware.com/mailman/listinfo/jemalloc-discuss
>>
>
>
>
> --
> Salvatore 'antirez' Sanfilippo
> open source developer - VMware
> http://invece.org
>
> Beauty is more important in computing than anywhere else in technology
> because software is so complicated. Beauty is the ultimate defence
> against complexity.
>        — David Gelernter