Enabling profiling causes hang?
Brock Pytlik
brock.pytlik at gmail.com
Wed Feb 11 14:49:59 PST 2015
So I think we're closing in on the finish line. I've successfully generated
profiles from ardb using the gate tip malloc. The problem I'm having is
that pprof (the one distributed w/ jemalloc) shows no allocation
information. (I use the one distributed w/ jemalloc b/c the one that's part
of gperftools 2.4 doesn't know about heap_v2 afaict.) To be clear, pprof
loads the file, then I do "top" and it says '0 B', as does "top --cum".
I haven't found documentation about what the heap_v2 file format is, so I'm
not sure where the problem is. I've included what one of the profile files
looks like. If you can point me to the documentation for the heap_v2 file
format that would be great, then I could possibly interpret what's below.
If you know what's going on when you look @ the file below, that would be
great as well.
heap_v2/524288
> t*: 217: 3029560 [0: 0]
> t0: 6: 1015808 [0: 0]
> t1: 0: 0 [0: 0]
> t2: 207: 1440312 [0: 0]
> t3: 0: 0 [0: 0]
> t4: 0: 0 [0: 0]
> t5: 0: 0 [0: 0]
> t6: 0: 0 [0: 0]
> t7: 4: 573440 [0: 0]
> @
> t*: 217: 3029560 [0: 0]
> t0: 6: 1015808 [0: 0]
> t2: 207: 1440312 [0: 0]
> t7: 4: 573440 [0: 0]
> MAPPED_LIBRARIES:
> 00008000-001f5000 r-xp 00000000 08:05 1318210 /usr/bin/ardb-server.prof
> 001fc000-00202000 rw-p 001ec000 08:05 1318210 /usr/bin/ardb-server.prof
> 00202000-00214000 rw-p 00000000 00:00 0 [heap]
> a6000000-adc00000 rw-p 00000000 00:00 0
> adc00000-adc01000 ---p 00000000 00:00 0
> adc01000-af400000 rw-p 00000000 00:00 0 [stack:5192]
> af400000-af401000 ---p 00000000 00:00 0
> af401000-b0400000 rw-p 00000000 00:00 0 [stack:4628]
> b0400000-b0401000 ---p 00000000 00:00 0
> b0401000-b1000000 rw-p 00000000 00:00 0 [stack:4627]
> b11ff000-b1200000 ---p 00000000 00:00 0
> b1200000-b19ff000 rw-p 00000000 00:00 0 [stack:4626]
> b19ff000-b1a00000 ---p 00000000 00:00 0
> b1a00000-b21ff000 rw-p 00000000 00:00 0 [stack:4625]
> b21ff000-b2200000 ---p 00000000 00:00 0
> b2200000-b29ff000 rw-p 00000000 00:00 0 [stack:4624]
> b29ff000-b2a00000 ---p 00000000 00:00 0
> b2a00000-b31ff000 rw-p 00000000 00:00 0 [stack:4623]
> b31ff000-b6400000 rw-s 00000000 08:06 2883593 <>/repl/repl.log
> b6400000-b6c00000 rw-p 00000000 00:00 0
> b6d25000-b6d26000 rw-p 00000000 00:00 0
> b6d27000-b6e42000 r-xp 00000000 08:05 130837 /lib/libc-2.13.so
> b6e42000-b6e49000 ---p 0011b000 08:05 130837 /lib/libc-2.13.so
> b6e49000-b6e4b000 r--p 0011a000 08:05 130837 /lib/libc-2.13.so
> b6e4b000-b6e4c000 rw-p 0011c000 08:05 130837 /lib/libc-2.13.so
> b6e4c000-b6e4f000 rw-p 00000000 00:00 0
> b6e4f000-b6e6e000 r-xp 00000000 08:05 130944 /lib/libgcc_s.so.1
> b6e6e000-b6e75000 ---p 0001f000 08:05 130944 /lib/libgcc_s.so.1
> b6e75000-b6e76000 rw-p 0001e000 08:05 130944 /lib/libgcc_s.so.1
> b6e76000-b6edf000 r-xp 00000000 08:05 130976 /lib/libm-2.13.so
> b6edf000-b6ee6000 ---p 00069000 08:05 130976 /lib/libm-2.13.so
> b6ee6000-b6ee7000 r--p 00068000 08:05 130976 /lib/libm-2.13.so
> b6ee7000-b6ee8000 rw-p 00069000 08:05 130976 /lib/libm-2.13.so
> b6ee8000-b6f84000 r-xp 00000000 08:05 1316222
> /usr/lib/libstdc++.so.6.0.17
> b6f84000-b6f8b000 ---p 0009c000 08:05 1316222
> /usr/lib/libstdc++.so.6.0.17
> b6f8b000-b6f8f000 r--p 0009b000 08:05 1316222
> /usr/lib/libstdc++.so.6.0.17
> b6f8f000-b6f91000 rw-p 0009f000 08:05 1316222
> /usr/lib/libstdc++.so.6.0.17
> b6f91000-b6f98000 rw-p 00000000 00:00 0
> b6f98000-b6fac000 r-xp 00000000 08:05 130941 /lib/libpthread-2.13.so
> b6fac000-b6fb3000 ---p 00014000 08:05 130941 /lib/libpthread-2.13.so
> b6fb3000-b6fb4000 r--p 00013000 08:05 130941 /lib/libpthread-2.13.so
> b6fb4000-b6fb5000 rw-p 00014000 08:05 130941 /lib/libpthread-2.13.so
> b6fb5000-b6fb7000 rw-p 00000000 00:00 0
> b6fb7000-b6fd4000 r-xp 00000000 08:05 130826 /lib/ld-2.13.so
> b6fd4000-b6fd7000 rw-p 00000000 00:00 0
> b6fd7000-b6fda000 rw-p 00000000 00:00 0
> b6fda000-b6fdb000 r-xp 00000000 00:00 0 [sigpage]
> b6fdb000-b6fdc000 r--p 0001c000 08:05 130826 /lib/ld-2.13.so
> b6fdc000-b6fdd000 rw-p 0001d000 08:05 130826 /lib/ld-2.13.so
> bef38000-bef59000 rw-p 00000000 00:00 0 [stack]
> ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors]
Thanks,
Brock
On Mon, Feb 9, 2015 at 3:23 PM, Jason Evans <jasone at canonware.com> wrote:
> On Feb 9, 2015, at 3:11 PM, Brock Pytlik <brock.pytlik at gmail.com> wrote:
>
> Well, I thought it would handle what I needed, but I can't seem to get it
> to produce a dump. Here's what I did:
> From my experiments on x86, it seems like prof:true needs to be part of
> MALLOC_CONF for lg_prof_interval to work at all. However, when I set
> prof:true on arm, ardb hangs in the atexit code. This happens even when I
> set prof_final to false. To be clear this command hangs (on arm):
>
>> MALLOC_CONF=prof:true,lg_prof_interval:2,prof_final:false ardb-server
>> ardb.conf
>
>
> Oh, whoops, I forgot that in the dev branch I refactored the prof
> bootstrapping code to make the atexit() call conditional, but in 3.6.0 the
> call always happens if prof:true is set. I think you will have to hack
> jemalloc in order to work around this.
>
> I'm confused about a couple of things:
> 1) Why it works on x86 and not arm (and whether that means the cross
> compiling build for arm has some issues).
>
>
> I haven't dug into this, but I'm assuming that this is an arbitrary
> implementation difference in the libc being used on ARM, possibly motivated
> by the desire to reduce the data segment size and only pay for internal
> atexit() metadata overhead if the functionality is actually used. I'm not
> very familiar with ARM-specific development, but I get the impression that
> there are multiple stripped down libc implementations in use.
>
> If these are things that are fixed in the gate tip of jemalloc, I'd also
> be happy to try changing ardb to use that (assuming it's largely a drop in
> replacement).
>
> Any suggestions on where I go next?
>
>
> Assuming that your current goal is simply to get heap profiles and
> diagnose a leak, I'd recommend just trying to use the dev version of
> jemalloc. If that doesn't work for some reason, then disabling the
> atexit() call in jemalloc's src/prof.c and rebuilding will probably be
> enough to get you what you need.
>
> Thanks,
> Jason
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jemalloc.net/mailman/jemalloc-discuss/attachments/20150211/0c761d33/attachment.html>
More information about the jemalloc-discuss
mailing list