Frequent segfaults in 2.2.5

[Rogier 'DocWilco' Mulhuijzen]

Upgraded libnuma, startup problem went away. We'll see if this improves
longterm stability.


On Tue, Jun 4, 2013 at 12:14 AM, Rogier 'DocWilco' Mulhuijzen <
rogier+jemalloc at fastly.com> wrote:

> We added --enable-fill and junk:true to the mix and ran headlong into a
> wall of libnuma init code. Looks like glibc malloc followed by jemalloc
> free. But even backporting the glibc hooks code from 3.x didn't help.
>
> We'll try 3.4 in the morning, see where that takes us.
>
> Thanks for the pointers so far. :)
>
> Cheers,
>
>       Doc
>
>
> On Mon, Jun 3, 2013 at 9:11 PM, Jason Evans <jasone at canonware.com> wrote:
>
>> On Jun 3, 2013, at 8:19 PM, Rogier 'DocWilco' Mulhuijzen <
>> rogier+jemalloc at fastly.com> wrote:
>>
>> We're currently using jemalloc 2.2.5 statically linked into a private
>> fork of Varnish with a very high rate of malloc/calloc/free, and we're
>> seeing segfaults on a somewhat frequent basis. (one a day on a group of 6
>> hosts.)
>>
>> We had the same segfaults with 2.2.3, and upgrading to 2.2.5 seems not to
>> have helped.
>>
>> (Also, we tried upgrading to 3.3.1 and things just got worse, tried
>> enabling debugging which made it even more worse. Under time pressure, we
>> dropped down to 2.2.5)
>>
>> I should mention that I backported the mmap strategy from 3.3.1 into
>> 2.2.5, to prevent VM fragmentation, which was causing us to run into
>> vm.max_map_count.
>>
>> So, to the meat of the problem! (We saw these in both 2.2.3 without the
>> mmap strategy backported, and 2.2.5 with mmap strategy backported.)
>>
>> Unfortunately, we don't have core files (we're running with 153G
>> resident, and 4075G virtual process size on one of the hosts that I'm
>> looking at right now) so the internal Varnish (libgcc based) backtrace is
>> all we have:
>>
>> *0x483894*: arena_tcache_fill_small+1a4
>> 0x4916b9: tcache_alloc_small_hard+19
>> 0x4841bf: arena_malloc+1bf
>> 0x47b498: calloc+218
>>
>> Looking that up:
>>
>> # addr2line -e /usr/sbin/varnishd -i 0x483894
>> /varnish-cache/lib/libjemalloc/include/jemalloc/internal/bitmap.h:101
>> /varnish-cache/lib/libjemalloc/include/jemalloc/internal/bitmap.h:140
>> /varnish-cache/lib/libjemalloc/src/arena.c:264
>> /varnish-cache/lib/libjemalloc/src/arena.c:1395
>>
>> Which looks like:
>>
>> 97 goff = bit >> LG_BITMAP_GROUP_NBITS;
>> 98 gp = &bitmap[goff];
>> 99 g = *gp;
>> 100 assert(g & (1LU << (bit & BITMAP_GROUP_NBITS_MASK)));
>> *101* g ^= 1LU << (bit & BITMAP_GROUP_NBITS_MASK);
>> 102 *gp = g;
>>
>> Which makes no sense at first, since there's no deref being done there,
>> but a disassembly (thanks Devon) shows:
>>
>>   483883:       48 c1 ef 06             shr    $0x6,%rdi
>>   483887:       83 e1 3f                and    $0x3f,%ecx
>>   48388a:       4c 8d 04 fa             lea    (%rdx,%rdi,8),%r8
>>   48388e:       49 d3 e1                shl    %cl,%r9
>>   483891:       4c 89 c9                mov    %r9,%rcx
>>   *483894*:       49 33 08                xor    (%r8),%rcx
>>
>> The optimizer got rid of g and just does the xor straight on *gp. So gp
>> is an illegal address. According to our segfault handler, it's NULL.
>>
>> For gp to be NULL, both bitmap and goff need to be NULL. And bitmap being
>> NULL is somewhat impossible due to:
>>
>> if ((run = bin->runcur) != NULL && run->nfree > 0)
>>  ptr = arena_run_reg_alloc(run, &arena_bin_info[binind]);
>>
>> bitmap is an offset to run, so both the offset and the run need to be
>> NULL (or perfectly matched to cancel eachother out, but also unlikely.)
>>
>> bin->runcur and bin->bitmap_offset both being NULL seems _very_ unlikely.
>>
>> And that's about as far as we've gotten.
>>
>> Help?
>>
>>
>> This sort of crash can happen as a distant result of a double-free.  For
>> example:
>>
>> 1) free(p)
>> 2) free(p) corrupts run counters, causing the run to be deallocated.
>> 3) free(q) causes the deallocated run to be placed back in service, but
>> with definite corruption.
>> 4) malloc(…) tries to allocate from run, but run metadata are in a bad
>> state.
>>
>> My suggestion is to enable assertions (something like: CFLAGS=-O3
>> ./configure --enable-debug --disable-tcache), disable tcache (which can
>> keep double free bugs from exercising the assertions), and look for the
>> source of corruption.  I'll be surprised if the problem deviates
>> substantially from the above, but if it does, then my next bet will be a
>> buffer overflow corrupting run metadata.
>>
>> Jason
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jemalloc.net/mailman/jemalloc-discuss/attachments/20130605/333118d2/attachment.html>