<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi All, <div> At Netapp, we use a modified version of FreeBSD for our user-space environment. </div><div><br></div><div>We've hit a bunch OOM problems where we ONLY have a core... Solving them USED to be</div><div>like wandering around in the dark... so I wrote some GDB pythons pretty printers to walk the heap and give breakdowns of how memory is used + </div><div>do some basic analysis of each piece of memory. </div><div><br></div><div>Those macros are attached and work on both cores & live processes on both FreeBSD and 'jemlloc_linux' version of jemalloc under linux. (32-bit/64-bit both work.)</div><div><br></div><div>Comments:</div><div>1) These macros are released under the same license as jemalloc. </div><div><br></div><div>2) They (currently) only support the jemalloc datastrutures of the FreeBSD version of jemalloc included in 7.X and 8.X FreeBSD</div><div> ===> (I haven't updated this to the latest jemalloc-2.X yet.) <====</div><div><br></div><div>3) I have unit-tests to verify that program allocations and gdb reported allocations match. </div><div><br></div><div> (If you're interested, we should figure how to integrate this with the jemalloc distribution.) </div><div><br></div><div>4) Fwiw.. These can be extended. </div><div><br></div><div> The internal functions can retrieve pointers to EVERY in-use piece of memory allocated through jemalloc. </div><div> RIght now, I use that to create the summary chart. Also, I can do some rudimentary analysis. (Is it a string?) </div><div> But.. motivated people could search for OTHER important data-structures.. </div><div><br></div><div>To Try on linux: </div><div>1) Build linux-jemalloc.</div><div>2) load gdb-enabled python with 'env LD_PRELOAD=../libjemalloc.so.0 gdb7.2 -q -x heap.py <YOUR APP>' (*)</div><div>3) run</div><div>4) br and type 'heap' </div><div><br></div><div>Help: </div><div><div><font class="Apple-style-span" face="Courier">(ugdb-amd64-7.2-02)help heap</font></div><div><font class="Apple-style-span" face="Courier">Show heap utilization for user-space cores and processes (BETA)</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> The code walks jemalloc heap datastructures to determine heap usage.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> It splits the memory usage into buckets by size.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> BETA: It is very memory intensive and can cause gdb to core.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> Example:</font></div><div><font class="Apple-style-span" face="Courier"> (ugdb-amd64-7.2-02) heap</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> FreeBSD Heap Analysis</font></div><div><font class="Apple-style-span" face="Courier"> ---------------------</font></div><div><font class="Apple-style-span" face="Courier"> Searching process virtual address space for 'chunks' of the heap... DONE.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> Calculating memory usage for 'chunks' in 5 heap arenas:</font></div><div><font class="Apple-style-span" face="Courier"> Walking Arena 0 (0x898f78) with 2 'chunks'... DONE.</font></div><div><font class="Apple-style-span" face="Courier"> Walking Arena 1 (0x89a6f8) with 0 'chunks'... DONE.</font></div><div><font class="Apple-style-span" face="Courier"> Walking Arena 2 (0x89b278) with 0 'chunks'... DONE.</font></div><div><font class="Apple-style-span" face="Courier"> Walking Arena 3 (0x899b78) with 19 'chunks'... DONE.</font></div><div><font class="Apple-style-span" face="Courier"> Walking Arena 4 (0x89bdf8) with 0 'chunks'... DONE.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> Heap allocation distribution (by size)</font></div><div><font class="Apple-style-span" face="Courier"> =================================================</font></div><div><font class="Apple-style-span" face="Courier"> 2 bytes * 96 allocs ( 0%) = 192 bytes ( 0%)</font></div><div><font class="Apple-style-span" face="Courier"> 4 bytes * 84 allocs ( 0%) = 336 bytes ( 0%)</font></div><div><font class="Apple-style-span" face="Courier"> 8 bytes * 2544 allocs ( 1%) = 20352 bytes ( 0%)</font></div><div><font class="Apple-style-span" face="Courier"> 16 bytes * 9569 allocs ( 4%) = 153104 bytes ( 0%)</font></div><div><font class="Apple-style-span" face="Courier"> 32 bytes * 47567 allocs (20%) = 1522144 bytes ( 9%)</font></div><div><font class="Apple-style-span" face="Courier"> ...</font></div><div><font class="Apple-style-span" face="Courier"> ^ ^ ^ ^ ^</font></div><div><font class="Apple-style-span" face="Courier"> | | | | |</font></div><div><font class="Apple-style-span" face="Courier"> Bin Size Num allocs % all allocs Total Bytes % of total bytes</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> Paper describing jemalloc: <a href="http://people.freebsd.org/~jasone/jemalloc/bsdcan2006/jemalloc.pdf">http://people.freebsd.org/~jasone/jemalloc/bsdcan2006/jemalloc.pdf</a></font></div><div><font class="Apple-style-span" face="Courier"> More jemalloc details: <a href="http://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919">http://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919</a></font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> 'heap stats [size]' => (default) Dump information about how the heap is used.</font></div><div><font class="Apple-style-span" face="Courier"> 'heap dump [size]' => Write all of the memory to appropriately sized files.</font></div><div><font class="Apple-style-span" face="Courier"> 'heap analyze [size]' => Try to the determine (and print) the objects located at the allocated memory.</font></div><div><font class="Apple-style-span" face="Courier"> 'heap examine [size]' => Examine region as if it was a bunch of pointers. (ie. 'x /2a 0xdeadbeef')</font></div></div><div><br></div><div>...</div><div><br></div><div>Hope you enjoy! </div><div><br></div><div><div>(FWIW. Here's an example running a gnome app, FWIW: 'env LD_PRELOAD=../libjemalloc.so.0 gdb7.2 -q -x /u/ezolt,spin/heap.py gedit') </div></div><div><br></div><div><br></div><div><font class="Apple-style-span" face="Courier">[ezolt@cyclptc11 test]$ env LD_PRELOAD=../libjemalloc.so.0 gdb7.2 -q -x /u/ezolt,spin/heap.py gedit</font></div><div><font class="Apple-style-span" face="Courier">Reading symbols from /usr/bin/gedit...(no debugging symbols found)...done.</font></div><div><font class="Apple-style-span" face="Courier">(gdb) run</font></div><div><font class="Apple-style-span" face="Courier">Starting program: /usr/bin/gedit </font></div><div><font class="Apple-style-span" face="Courier">[Thread debugging using libthread_db enabled]</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">Program received signal SIGTSTP, Stopped (user).</font></div><div><font class="Apple-style-span" face="Courier">0x00000038104cae0f in poll () from /lib64/libc.so.6</font></div><div><font class="Apple-style-span" face="Courier">(gdb) heap</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">Jemalloc Heap Analysis</font></div><div><font class="Apple-style-span" face="Courier">----------------------</font></div><div><font class="Apple-style-span" face="Courier">Searching process virtual address space for 'chunks' of the heap... DONE.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">Calculating memory usage details for each 'chunks' in 1 heap arenas: </font></div><div><font class="Apple-style-span" face="Courier"> Walking Arena 0 (0x2aaaaacb9040) with 11 'chunks'... DONE.</font></div><div><font class="Apple-style-span" face="Courier"> Walking global 'Huge' allocations... DONE.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">Heap allocation distribution (by size)</font></div><div><font class="Apple-style-span" face="Courier">=================================================</font></div><div><font class="Apple-style-span" face="Courier"> 2 bytes * 276 allocs ( 0%) = 552 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 4 bytes * 513 allocs ( 1%) = 2,052 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 8 bytes * 1,458 allocs ( 4%) = 11,664 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 16 bytes * 5,493 allocs (15%) = 87,888 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 32 bytes * 12,237 allocs (34%) = 391,584 bytes ( 4%) </font></div><div><font class="Apple-style-span" face="Courier"> 48 bytes * 4,061 allocs (11%) = 194,928 bytes ( 2%) </font></div><div><font class="Apple-style-span" face="Courier"> 64 bytes * 4,076 allocs (11%) = 260,864 bytes ( 2%) </font></div><div><font class="Apple-style-span" face="Courier"> 80 bytes * 1,282 allocs ( 3%) = 102,560 bytes ( 1%) </font></div><div><font class="Apple-style-span" face="Courier"> 96 bytes * 596 allocs ( 1%) = 57,216 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 112 bytes * 185 allocs ( 0%) = 20,720 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 128 bytes * 119 allocs ( 0%) = 15,232 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 192 bytes * 433 allocs ( 1%) = 83,136 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 256 bytes * 283 allocs ( 0%) = 72,448 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 320 bytes * 1,217 allocs ( 3%) = 389,440 bytes ( 4%) </font></div><div><font class="Apple-style-span" face="Courier"> 384 bytes * 73 allocs ( 0%) = 28,032 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 448 bytes * 47 allocs ( 0%) = 21,056 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 512 bytes * 1,210 allocs ( 3%) = 619,520 bytes ( 6%) </font></div><div><font class="Apple-style-span" face="Courier"> 768 bytes * 670 allocs ( 1%) = 514,560 bytes ( 5%) </font></div><div><font class="Apple-style-span" face="Courier"> 1024 bytes * 655 allocs ( 1%) = 670,720 bytes ( 6%) </font></div><div><font class="Apple-style-span" face="Courier"> 1280 bytes * 42 allocs ( 0%) = 53,760 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 1536 bytes * 19 allocs ( 0%) = 29,184 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 1792 bytes * 23 allocs ( 0%) = 41,216 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 2048 bytes * 80 allocs ( 0%) = 163,840 bytes ( 1%) </font></div><div><font class="Apple-style-span" face="Courier"> 2304 bytes * 10 allocs ( 0%) = 23,040 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 2560 bytes * 14 allocs ( 0%) = 35,840 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 2816 bytes * 4 allocs ( 0%) = 11,264 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 3072 bytes * 128 allocs ( 0%) = 393,216 bytes ( 4%) </font></div><div><font class="Apple-style-span" face="Courier"> 3328 bytes * 1 allocs ( 0%) = 3,328 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 3584 bytes * 6 allocs ( 0%) = 21,504 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 3840 bytes * 1 allocs ( 0%) = 3,840 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 4096 bytes * 42 allocs ( 0%) = 172,032 bytes ( 1%) </font></div><div><font class="Apple-style-span" face="Courier">---Type <return> to continue, or q <return> to quit---</font></div><div><font class="Apple-style-span" face="Courier"> 8192 bytes * 34 allocs ( 0%) = 278,528 bytes ( 2%) </font></div><div><font class="Apple-style-span" face="Courier"> 12288 bytes * 50 allocs ( 0%) = 614,400 bytes ( 6%) </font></div><div><font class="Apple-style-span" face="Courier"> 16384 bytes * 16 allocs ( 0%) = 262,144 bytes ( 2%) </font></div><div><font class="Apple-style-span" face="Courier"> 20480 bytes * 1 allocs ( 0%) = 20,480 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 24576 bytes * 1 allocs ( 0%) = 24,576 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 28672 bytes * 1 allocs ( 0%) = 28,672 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 45056 bytes * 2 allocs ( 0%) = 90,112 bytes ( 0%) </font></div><div><font class="Apple-style-span" face="Courier"> 49152 bytes * 3 allocs ( 0%) = 147,456 bytes ( 1%) </font></div><div><font class="Apple-style-span" face="Courier"> 65536 bytes * 10 allocs ( 0%) = 655,360 bytes ( 6%) </font></div><div><font class="Apple-style-span" face="Courier"> 98304 bytes * 1 allocs ( 0%) = 98,304 bytes ( 1%) </font></div><div><font class="Apple-style-span" face="Courier"> 262144 bytes * 8 allocs ( 0%) = 2,097,152 bytes (21%) </font></div><div><font class="Apple-style-span" face="Courier"> 786432 bytes * 1 allocs ( 0%) = 786,432 bytes ( 8%) </font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">Heap: 35,382 allocs (9,599,852 bytes) </font></div><div><font class="Apple-style-span" face="Courier">Total VM allocated by heap: 11,534,336 bytes (In-use by application: 9,599,852)</font></div><div><font class="Apple-style-span" face="Courier">(gdb) quit</font></div><div><font class="Apple-style-span" face="Courier">A debugging session is active.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"><span class="Apple-tab-span" style="white-space:pre"> </span>Inferior 1 [process 7804] will be killed.</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">Quit anyway? (y or n) y </font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div>Cheers,</div><div>--Phil</div><div><br></div><div></div></body></html>