<div dir="ltr">Well, I thought it would handle what I needed, but I can't seem to get it to produce a dump. Here's what I did:<div>From my experiments on x86, it seems like prof:true needs to be part of MALLOC_CONF for lg_prof_interval to work at all. However, when I set prof:true on arm, ardb hangs in the atexit code. This happens even when I set prof_final to false. To be clear this command hangs (on arm):</div><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">MALLOC_CONF=prof:true,lg_prof_interval:2,prof_final:false ardb-server ardb.conf</blockquote><div> </div></div><div>This command works, but doesn't produce any dumps (on x86 or arm):</div><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">MALLOC_CONF=lg_prof_interval:2,prof_final:false ardb-server ./httpd/ardb/ardb.conf </blockquote></div><div><br></div><div>I even tried:</div><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">MALLOC_CONF=prof:true,prof_gdump:false,prof_active:false,lg_prof_interval:2,prof_final:false,prof_leak:false ardb-server ardb.conf</blockquote></div><div>but that also hung (on arm).</div><div><br></div><div>I'm confused about a couple of things:</div><div>1) Why it works on x86 and not arm (and whether that means the cross compiling build for arm has some issues).</div><div>2) Why it hangs on arm in the atexit thing even when prof_final (and the other profiling flags) are set to false.</div><div><br></div><div>If these are things that are fixed in the gate tip of jemalloc, I'd also be happy to try changing ardb to use that (assuming it's largely a drop in replacement).</div><div><br></div><div>Any suggestions on where I go next?</div><div><br></div><div>Thanks,</div><div>Brock</div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 7, 2015 at 12:17 PM, Brock Pytlik <span dir="ltr"><<a href="mailto:brock.pytlik@gmail.com" target="_blank">brock.pytlik@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Awesome, thanks for getting back to me so quickly. I wasn't understanding quite how the lg_prof_interval option was working. That should definitely handle what I need!<span class="HOEnZb"><font color="#888888"><div><br></div><div>Brock</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 6, 2015 at 10:47 PM, Jason Evans <span dir="ltr"><<a href="mailto:jasone@canonware.com" target="_blank">jasone@canonware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>On Feb 6, 2015, at 6:40 PM, Brock Pytlik <<a href="mailto:brock.pytlik@gmail.com" target="_blank">brock.pytlik@gmail.com</a>> wrote:<br>
> I'm trying to track down a possible memory leak in ardb, which uses jemalloc, when running on an arm processor.<br>
><br>
> The problem I have is that when I set 'MALLOC_CONF=prof_leak:true' and start ardb, the program hangs.<br>
> When I strace it, this is the system call it does:<br>
> futex(0xb6db3c0c, FUTEX_WAIT_PRIVATE, 2, NULL<br>
><br>
> Here's the stack trace gdb reports:<br>
> (gdb) bt<br>
> #0 0xb6e1af4c in __lll_lock_wait_private () from /lib/libc.so.6<br>
> #1 0xb6d79de4 in __new_exitfn () from /lib/libc.so.6<br>
> #2 0xb6d79e34 in __internal_atexit () from /lib/libc.so.6<br>
> #3 0x0016d75c in je_prof_boot2 () at src/prof.c:1349<br>
> #4 0x00143418 in malloc_init_hard () at src/jemalloc.c:767<br>
> #5 0x0014db70 in malloc_init () at src/jemalloc.c:292<br>
> #6 calloc (num=1, size=520) at src/jemalloc.c:1123<br>
> #7 0xb6d79da4 in __new_exitfn () from /lib/libc.so.6<br>
> #8 0xb6d79e34 in __internal_atexit () from /lib/libc.so.6<br>
> #9 0x00013410 in __static_initialization_and_destruction_0 (<br>
> __initialize_p=<optimized out>, __priority=<optimized out>)<br>
> at <>/include/c++/4.7.3/iostream:75<br>
> #10 _GLOBAL__sub_I_channel.cpp(void) () at common/channel/channel.cpp:738<br>
> #11 0x001786f8 in __libc_csu_init ()<br>
> #12 0xb6d61834 in __libc_start_main () from /lib/libc.so.6<br>
> #13 0x0001424c in _start ()<br>
><br>
> I'll admit at this point I'm kind of stumped as to how to proceed. I thought I'd start by asking here in case anyone had seen similar behavior or knew what the problem was. I checked the issues at the github page and couldn't find anything similar. This is using version 3.6.0 of jemalloc.<br>
><br>
> When I looked at the jemalloc.xml user manual in the gate tip (not the 3.6.0 branch I'm using) I saw some discussion of atexit being problematic in the context of prof_final. If that's the problem, is there a way to generate profiling information either while the program is running, avoiding the atexit issue? I didn't quite follow the comment in the user manual that says that an application can register its own atexit parameter with equivalent functionality. Is there an example someplace I could crib from or could someone help me understand that a bit?<br>
<br>
</div></div>Yes, it looks like you're hitting the same atexit() issue that caused me to disable prof_final by default in the dev branch of jemalloc. If you instead use MALLOC_CONF=prof_final:false,lg_prof_interval=20 (see <a href="http://www.canonware.com/download/jemalloc/jemalloc-latest/doc/jemalloc.html#opt.lg_prof_interval" target="_blank">http://www.canonware.com/download/jemalloc/jemalloc-latest/doc/jemalloc.html#opt.lg_prof_interval</a> and choose an appropriate dump interval), you should be able to get useful heap profiles without modifying the ardb source code. The prof_gdump option provides an alternative mechanism for triggering heap dumps. If you want to go as far as modifying ardb source code, you can call the prof.dump mallctl to trigger a dump. None of these options will cause jemalloc to print a leak report during exit, but the resulting heap profiles are what you really need anyway.<br>
<span><font color="#888888"><br>
Jason</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>