<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <style type="text/css" style="display:none"></style> </head> <body dir="ltr"> <div id="OWAFontStyleDivID" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;"> <p>Hello,</p> <p><br> </p> <p>Some time ago I ported jemalloc to work on old systems still using LinuxThreads. See this thread: <a href="http://jemalloc.net/mailman/jemalloc-discuss/2013-October/000646.html" id="lnk281466">http://jemalloc.net/mailman/jemalloc-discuss/2013-October/000646.html</a><br> </p> <p><br> </p> <p>It seems that with LinuxThreads the fork() implementation will call free during fork, between jemalloc_prefork and jemalloc_postfork_child. This can sometimes (very rarely) hang in a call to system() with a stack like the following:<br> </p> <p><br> </p> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#0 0x0fdd660c in __pthread_sigsuspend () from /lib/libpthread.so.0</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#1 0x0fdd6344 in __pthread_wait_for_restart_signal () from /lib/libpthread.so.0</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#2 0x0fdd805c in __pthread_alt_lock () from /lib/libpthread.so.0</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#3 0x0fdd4c74 in pthread_mutex_lock () from /lib/libpthread.so.0</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#4 0x0ff2a480 in pthread_mutex_lock () from /lib/libc.so.6</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#5 0x0ffc5bf0 in malloc_mutex_lock (tbin=0x3082c020, binind=0, rem=0, tcache=0x3082c000) at ../../src/jemalloc-3.0.0/include/jemalloc/internal/mutex.h:77</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#6 tcache_bin_flush_small (tbin=0x3082c020, binind=0, rem=0, tcache=0x3082c000) at ../../src/jemalloc-3.0.0/src/tcache.c:106</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#7 0x0ffc64c0 in tcache_event_hard (tcache=0x3082c000) at ../../src/jemalloc-3.0.0/src/tcache.c:39</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;"><strong>#8 0x0ffa545c in tcache_event (ptr=0x3081e000) at ../../src/jemalloc-3.0.0/include/jemalloc/internal/tcache.h:271</strong></span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#9 tcache_dalloc_large (ptr=0x3081e000) at ../../src/jemalloc-3.0.0/include/jemalloc/internal/tcache.h:435</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#10 arena_dalloc (ptr=0x3081e000) at ../../src/jemalloc-3.0.0/include/jemalloc/internal/arena.h:966</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#11 idalloc (ptr=0x3081e000) at include/jemalloc/internal/jemalloc_internal.h:840</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#12 iqalloc (ptr=0x3081e000) at include/jemalloc/internal/jemalloc_internal.h:852</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#13 free (ptr=0x3081e000) at ../../src/jemalloc-3.0.0/src/jemalloc.c:1219</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#14 0x0fdd6174 in __pthread_reset_main_thread () from /lib/libpthread.so.0</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#15 0x0fdd5288 in __pthread_fork () from /lib/libpthread.so.0</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#16 0x0feeadc4 in fork () from /lib/libc.so.6</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#17 0x0fe82eb0 in do_system () from /lib/libc.so.6</span></div> <div><span style="font-family: Consolas, monospace; font-size: 11pt;">#18 0x0fe830c8 in system () from /lib/libc.so.6</span><br> </div> <div><br> </div> <div>I am not familiar with how jemalloc works internally but it seems that sometimes tcache_event will trigger some sort of GC. Sometimes (very rarely) this attempts to take a lock which is already taken inside jemalloc_prefork. This hangs because locks are not recursive by default.<br> </div> <div><br> </div> <div>I was able to reproduce this in a standalone program. The issue seems to go away if I avoid the GC as in the attached patch. It seems like a horrible evil hack.<br> </div> <div> <ul> <li><span style="font-size: 12pt;">Am I missing anything? <span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">A</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">re there any other </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">p</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">latforms</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);"> with this issue?</span></span></li><li><span style="font-size: 12pt;">Can you see something else going wrong because of free-during-fork? I could patch jemalloc to leak that memory. I only really care about system(), not other uses of fork().</span><br> </li><li><span style="font-size: 12pt;">Can you think of a cleaner solution?</span></li></ul> <div>The patch is on top of 3.0.0 so it won't apply cleanly. I tried to apply commits <a href="https://github.com/jemalloc/jemalloc/commit/20f1fc95adb35ea63dc61f47f2b0ffbd37d39f32">20f1fc95adb35ea63dc61f47f2b0ffbd37d39f32</a> and <a href="https://github.com/jemalloc/jemalloc/commit/b5225928fe106a7d809bd34e849abcd6941e93c7">b5225928fe106a7d809bd34e849abcd6941e93c7</a> but they did not help me.<br> </div> <div><br> </div> <div>Regards,<br> </div> </div> <div>Leonard<br> </div> </div> </body> </html>