Need help with patch allowing to iterate over all objects

[Benoit Jacob]

In the assertion failure in my last email, I have these local variables:

(gdb) p run->nfree
$1 = 32767
(gdb) p bin_info->nregs
$2 = 51

Benoit

2012/8/24 Benoit Jacob <jacob.benoit.1 at gmail.com>:
> And another crash, also an assertion about nfree not having the expected value:
>
> <jemalloc>: /hack/mozilla-central/memory/jemalloc/src/src/arena.c:165:
> Failed assertion: "run->nfree < bin_info->nregs"
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000411605 in moz_abort () at
> /hack/mozilla-central/memory/build/extraMallocFuncs.c:116
> 116       MOZ_CRASH();
> (gdb) bt
> #0  0x0000000000411605 in moz_abort () at
> /hack/mozilla-central/memory/build/extraMallocFuncs.c:116
> #1  0x000000000041743a in arena_run_reg_dalloc (run=0x7fffbe37f000,
> ptr=0x7fffbe38af00) at
> /hack/mozilla-central/memory/jemalloc/src/src/arena.c:165
> #2  0x000000000041c5e7 in arena_dalloc_bin_locked
> (arena=0x7ffff6c00180, chunk=0x7fffbe300000, ptr=0x7fffbe38af00,
> mapelm=0x7fffbe300ce8) at
> /hack/mozilla-central/memory/jemalloc/src/src/arena.c:1590
> #3  0x000000000043c583 in tcache_bin_flush_small (tbin=0x7ffff6b022c8,
> binind=21, rem=51, tcache=0x7ffff6b02000) at
> /hack/mozilla-central/memory/jemalloc/src/src/tcache.c:128
> #4  0x000000000043bdd3 in tcache_dalloc_small (tcache=0x7ffff6b02000,
> ptr=0x7fffca24ff00, binind=21) at
> /hack/mozilla-central/memory/jemalloc/src/include/jemalloc/internal/tcache.h:399
> #5  0x00000000004132e8 in arena_dalloc (arena=0x7ffff6c00180,
> chunk=0x7fffca200000, ptr=0x7fffca24ff00, try_tcache=true) at
> /hack/mozilla-central/memory/jemalloc/src/include/jemalloc/internal/arena.h:956
> #6  0x000000000042e13d in idalloc (ptr=0x7fffca24ff00) at
> src/include/jemalloc/internal/jemalloc_internal.h:840
> #7  0x000000000042e18f in iqalloc (ptr=0x7fffca24ff00) at
> src/include/jemalloc/internal/jemalloc_internal.h:852
> #8  0x0000000000432a58 in real_je_free (ptr=0x7fffca24ff00) at
> /hack/mozilla-central/memory/jemalloc/src/src/jemalloc.c:1212
> #9  0x00000000004330d8 in free (ptr=0x7fffca24ff20) at
> /hack/mozilla-central/memory/jemalloc/src/src/jemalloc.c:1458
> #10 0x00007ffff7fe8022 in moz_free (ptr=0x7fffca24ff20) at
> /hack/mozilla-central/memory/mozalloc/mozalloc.cpp:51
> #11 0x00007ffff472490c in nsStringBuffer::Release
> (this=0x7fffca24ff20) at
> /hack/mozilla-central/xpcom/string/src/nsSubstring.cpp:161
> #12 0x00007ffff3e4c777 in FinalizeDOMString (fin=0x7ffff64b5058,
> chars=0x7fffca24ff28) at
> /hack/mozilla-central/js/xpconnect/src/XPCString.cpp:27
> #13 0x00007ffff514a137 in JSExternalString::finalize
> (this=0x7fffbf38d6c0, fop=0x7fffffffa000) at
> /hack/mozilla-central/js/src/vm/String-inl.h:439
> #14 0x00007ffff515f2a8 in js::gc::Arena::finalize<JSExternalString>
> (this=0x7fffbf38d000, fop=0x7fffffffa000,
> thingKind=js::gc::FINALIZE_EXTERNAL_STRING, thingSize=32) at
> /hack/mozilla-central/js/src/jsgc.cpp:319
>
>
> What can I do to help you help me?
>
> Could anyone at least give me a quick explanation of what these
> assertions are about?
>
> Thanks!
> Benoit
>
>
> 2012/8/21 Benoit Jacob <jacob.benoit.1 at gmail.com>:
>> Hi,
>>
>> I am attaching a patch that allows to iterate over all objects.
>>
>> This is not a request for upstreaming: these patches cause crashes and I
>> need help understanding these. And even if it didn't crash, this patch
>> causes a severe overhead in memory usage and no attempt has been made to
>> mitigate it. The goal of this effort is to allow custom developer builds of
>> Firefox to introspect their own allocated blocks.
>>
>> The patch (attached to this email) works by renaming the public functions
>> such as je_malloc() to real_je_malloc() and implementing custom je_malloc()
>> to allocate a larger block, and use the extra space to store the data of a
>> doubly-linked list element.
>>
>> In principle, this should be entirely transparent to the user (except for
>> the increased memory usage), so I don't understand the crashes.
>>
>> The crashes aren't immediate: I can run Firefox (this patch if for Firefox's
>> copy of jemalloc 3.0) for a while and browse a few pages without crashing.
>>
>> The crashes are assertion failures like this:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x0000000000411605 in moz_abort ()
>>     at /hack/mozilla-central/memory/build/extraMallocFuncs.c:116
>> 116       MOZ_CRASH();
>> (gdb) bt
>> #0  0x0000000000411605 in moz_abort ()
>>     at /hack/mozilla-central/memory/build/extraMallocFuncs.c:116
>> #1  0x000000000041afdf in arena_bin_malloc_hard (arena=0x7ffff6c00180,
>> bin=0x7ffff6c007c8)
>>     at /hack/mozilla-central/memory/jemalloc/src/src/arena.c:1189
>> #2  0x000000000041b225 in arena_tcache_fill_small (arena=0x7ffff6c00180,
>>     tbin=0x7ffff6b02148, binind=9, prof_accumbytes=0)
>>     at /hack/mozilla-central/memory/jemalloc/src/src/arena.c:1232
>> #3  0x000000000043c239 in tcache_alloc_small_hard (tcache=0x7ffff6b02000,
>>     tbin=0x7ffff6b02148, binind=9)
>>     at /hack/mozilla-central/memory/jemalloc/src/src/tcache.c:72
>> #4  0x000000000043b7dc in tcache_alloc_small (tcache=0x7ffff6b02000,
>> size=160, zero=false)
>>     at
>> /hack/mozilla-central/memory/jemalloc/src/include/jemalloc/internal/tcache.h:302
>> #5  0x0000000000412c26 in arena_malloc (arena=0x0, size=160, zero=false,
>> try_tcache=true)
>>     at
>> /hack/mozilla-central/memory/jemalloc/src/include/jemalloc/internal/arena.h:869
>> #6  0x000000000042dce8 in imalloc (size=160)
>>     at src/include/jemalloc/internal/jemalloc_internal.h:735
>> #7  0x000000000043111c in real_je_malloc (size=160)
>>     at /hack/mozilla-central/memory/jemalloc/src/src/jemalloc.c:829
>> #8  0x0000000000432fa3 in malloc (size=128)
>>     at /hack/mozilla-central/memory/jemalloc/src/src/jemalloc.c:1425
>> #9  0x00007ffff7fe803c in moz_xmalloc (size=128)
>>     at /hack/mozilla-central/memory/mozalloc/mozalloc.cpp:57
>> #10 0x00007ffff2dcb74d in nsTArrayInfallibleAllocator::Malloc (size=128)
>>     at ../../dist/include/nsTArray.h:56
>> ...
>>
>> Printing some variables here:
>>
>> (gdb) up
>> #1  0x000000000041afdf in arena_bin_malloc_hard (arena=0x7ffff6c00180,
>> bin=0x7ffff6c007c8)
>>     at /hack/mozilla-central/memory/jemalloc/src/src/arena.c:1189
>> 1189            assert(bin->runcur->nfree > 0);
>> (gdb) p bin
>> $1 = (arena_bin_t *) 0x7ffff6c007c8
>> (gdb) p *bin
>> $2 = {lock = {lock = {__data = {__lock = 1, __count = 0, __owner = 6469,
>> __nusers = 1,
>>         __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
>>       __size = "\001\000\000\000\000\000\000\000E\031\000\000\001", '\000'
>> <repeats 26 times>, __align = 1}}, runcur = 0x7fffc2ced000, runs = {rbt_root
>> = 0x7fffca3004d8, rbt_nil = {{
>>         u = {rb_link = {rbn_left = 0x7ffff6c00800, rbn_right_red =
>> 0x7ffff6c00800},
>>           ql_link = {qre_next = 0x7ffff6c00800, qre_prev = 0x7ffff6c00800}},
>>         prof_ctx = 0x7ffff6c00800}, bits = 0}}, stats = {allocated =
>> 8896000,
>>     nmalloc = 197757, ndalloc = 142157, nrequests = 300344, nfills = 5545,
>>     nflushes = 2848, nruns = 2802, reruns = 5838, curruns = 1143}}
>> (gdb) p bin->runcur
>> $3 = (arena_run_t *) 0x7fffc2ced000
>> (gdb) p *(bin->runcur)
>> $4 = {bin = 0x7ffff6c007c8, nextind = 4544384, nfree = 0}
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>> Benoit