<div dir="ltr">So I think we're closing in on the finish line. I've successfully generated profiles from ardb using the gate tip malloc. The problem I'm having is that pprof (the one distributed w/ jemalloc) shows no allocation information. (I use the one distributed w/ jemalloc b/c the one that's part of gperftools 2.4 doesn't know about heap_v2 afaict.) To be clear, pprof loads the file, then I do "top" and it says '0 B', as does "top --cum".<div><br></div><div>I haven't found documentation about what the heap_v2 file format is, so I'm not sure where the problem is. I've included what one of the profile files looks like. If you can point me to the documentation for the heap_v2 file format that would be great, then I could possibly interpret what's below. If you know what's going on when you look @ the file below, that would be great as well.</div><div><br></div><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">heap_v2/524288<br> t*: 217: 3029560 [0: 0]<br> t0: 6: 1015808 [0: 0]<br> t1: 0: 0 [0: 0]<br> t2: 207: 1440312 [0: 0]<br> t3: 0: 0 [0: 0]<br> t4: 0: 0 [0: 0]<br> t5: 0: 0 [0: 0]<br> t6: 0: 0 [0: 0]<br> t7: 4: 573440 [0: 0]<br>@<br> t*: 217: 3029560 [0: 0]<br> t0: 6: 1015808 [0: 0]<br> t2: 207: 1440312 [0: 0]<br> t7: 4: 573440 [0: 0]<br>MAPPED_LIBRARIES:<br>00008000-001f5000 r-xp 00000000 08:05 1318210 /usr/bin/ardb-server.prof<br>001fc000-00202000 rw-p 001ec000 08:05 1318210 /usr/bin/ardb-server.prof<br>00202000-00214000 rw-p 00000000 00:00 0 [heap]<br>a6000000-adc00000 rw-p 00000000 00:00 0 <br>adc00000-adc01000 ---p 00000000 00:00 0 <br>adc01000-af400000 rw-p 00000000 00:00 0 [stack:5192]<br>af400000-af401000 ---p 00000000 00:00 0 <br>af401000-b0400000 rw-p 00000000 00:00 0 [stack:4628]<br>b0400000-b0401000 ---p 00000000 00:00 0 <br>b0401000-b1000000 rw-p 00000000 00:00 0 [stack:4627]<br>b11ff000-b1200000 ---p 00000000 00:00 0 <br>b1200000-b19ff000 rw-p 00000000 00:00 0 [stack:4626]<br>b19ff000-b1a00000 ---p 00000000 00:00 0 <br>b1a00000-b21ff000 rw-p 00000000 00:00 0 [stack:4625]<br>b21ff000-b2200000 ---p 00000000 00:00 0 <br>b2200000-b29ff000 rw-p 00000000 00:00 0 [stack:4624]<br>b29ff000-b2a00000 ---p 00000000 00:00 0 <br>b2a00000-b31ff000 rw-p 00000000 00:00 0 [stack:4623]<br>b31ff000-b6400000 rw-s 00000000 08:06 2883593 <>/repl/repl.log<br>b6400000-b6c00000 rw-p 00000000 00:00 0 <br>b6d25000-b6d26000 rw-p 00000000 00:00 0 <br>b6d27000-b6e42000 r-xp 00000000 08:05 130837 /lib/<a href="http://libc-2.13.so">libc-2.13.so</a><br>b6e42000-b6e49000 ---p 0011b000 08:05 130837 /lib/<a href="http://libc-2.13.so">libc-2.13.so</a><br>b6e49000-b6e4b000 r--p 0011a000 08:05 130837 /lib/<a href="http://libc-2.13.so">libc-2.13.so</a><br>b6e4b000-b6e4c000 rw-p 0011c000 08:05 130837 /lib/<a href="http://libc-2.13.so">libc-2.13.so</a><br>b6e4c000-b6e4f000 rw-p 00000000 00:00 0 <br>b6e4f000-b6e6e000 r-xp 00000000 08:05 130944 /lib/libgcc_s.so.1<br>b6e6e000-b6e75000 ---p 0001f000 08:05 130944 /lib/libgcc_s.so.1<br>b6e75000-b6e76000 rw-p 0001e000 08:05 130944 /lib/libgcc_s.so.1<br>b6e76000-b6edf000 r-xp 00000000 08:05 130976 /lib/<a href="http://libm-2.13.so">libm-2.13.so</a><br>b6edf000-b6ee6000 ---p 00069000 08:05 130976 /lib/<a href="http://libm-2.13.so">libm-2.13.so</a><br>b6ee6000-b6ee7000 r--p 00068000 08:05 130976 /lib/<a href="http://libm-2.13.so">libm-2.13.so</a><br>b6ee7000-b6ee8000 rw-p 00069000 08:05 130976 /lib/<a href="http://libm-2.13.so">libm-2.13.so</a><br>b6ee8000-b6f84000 r-xp 00000000 08:05 1316222 /usr/lib/libstdc++.so.6.0.17<br>b6f84000-b6f8b000 ---p 0009c000 08:05 1316222 /usr/lib/libstdc++.so.6.0.17<br>b6f8b000-b6f8f000 r--p 0009b000 08:05 1316222 /usr/lib/libstdc++.so.6.0.17<br>b6f8f000-b6f91000 rw-p 0009f000 08:05 1316222 /usr/lib/libstdc++.so.6.0.17<br>b6f91000-b6f98000 rw-p 00000000 00:00 0 <br>b6f98000-b6fac000 r-xp 00000000 08:05 130941 /lib/<a href="http://libpthread-2.13.so">libpthread-2.13.so</a><br>b6fac000-b6fb3000 ---p 00014000 08:05 130941 /lib/<a href="http://libpthread-2.13.so">libpthread-2.13.so</a><br>b6fb3000-b6fb4000 r--p 00013000 08:05 130941 /lib/<a href="http://libpthread-2.13.so">libpthread-2.13.so</a><br>b6fb4000-b6fb5000 rw-p 00014000 08:05 130941 /lib/<a href="http://libpthread-2.13.so">libpthread-2.13.so</a><br>b6fb5000-b6fb7000 rw-p 00000000 00:00 0 <br>b6fb7000-b6fd4000 r-xp 00000000 08:05 130826 /lib/<a href="http://ld-2.13.so">ld-2.13.so</a><br>b6fd4000-b6fd7000 rw-p 00000000 00:00 0 <br>b6fd7000-b6fda000 rw-p 00000000 00:00 0 <br>b6fda000-b6fdb000 r-xp 00000000 00:00 0 [sigpage]<br>b6fdb000-b6fdc000 r--p 0001c000 08:05 130826 /lib/<a href="http://ld-2.13.so">ld-2.13.so</a><br>b6fdc000-b6fdd000 rw-p 0001d000 08:05 130826 /lib/<a href="http://ld-2.13.so">ld-2.13.so</a><br>bef38000-bef59000 rw-p 00000000 00:00 0 [stack]<br>ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors]</blockquote>
</div><div><br></div><div><br></div><div>Thanks,</div><div>Brock</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 9, 2015 at 3:23 PM, Jason Evans <span dir="ltr"><<a href="mailto:jasone@canonware.com" target="_blank">jasone@canonware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><span class="">On Feb 9, 2015, at 3:11 PM, Brock Pytlik <<a href="mailto:brock.pytlik@gmail.com" target="_blank">brock.pytlik@gmail.com</a>> wrote:</span><div><span class=""><blockquote type="cite"><div><div dir="ltr">Well, I thought it would handle what I needed, but I can't seem to get it to produce a dump. Here's what I did:<div>From my experiments on x86, it seems like prof:true needs to be part of MALLOC_CONF for lg_prof_interval to work at all. However, when I set prof:true on arm, ardb hangs in the atexit code. This happens even when I set prof_final to false. To be clear this command hangs (on arm):</div><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">MALLOC_CONF=prof:true,lg_prof_interval:2,prof_final:false ardb-server ardb.conf</blockquote></div></div></div></blockquote><div><br></div></span><div>Oh, whoops, I forgot that in the dev branch I refactored the prof bootstrapping code to make the atexit() call conditional, but in 3.6.0 the call always happens if prof:true is set. I think you will have to hack jemalloc in order to work around this.</div><span class=""><br><blockquote type="cite"><div dir="ltr"><div>I'm confused about a couple of things:</div><div>1) Why it works on x86 and not arm (and whether that means the cross compiling build for arm has some issues).</div></div></blockquote><div><br></div></span><div>I haven't dug into this, but I'm assuming that this is an arbitrary implementation difference in the libc being used on ARM, possibly motivated by the desire to reduce the data segment size and only pay for internal atexit() metadata overhead if the functionality is actually used. I'm not very familiar with ARM-specific development, but I get the impression that there are multiple stripped down libc implementations in use.</div><span class=""><br><blockquote type="cite"><div dir="ltr"><div>If these are things that are fixed in the gate tip of jemalloc, I'd also be happy to try changing ardb to use that (assuming it's largely a drop in replacement).</div><div><br></div><div>Any suggestions on where I go next?</div></div></blockquote></span></div><br><div>Assuming that your current goal is simply to get heap profiles and diagnose a leak, I'd recommend just trying to use the dev version of jemalloc. If that doesn't work for some reason, then disabling the atexit() call in jemalloc's src/prof.c and rebuilding will probably be enough to get you what you need.</div><div><br></div><div>Thanks,</div><div>Jason</div></div></blockquote></div><br></div>