Hi Jason, Thx for taking the time to look into this ! Below is the requested information. When I originally observed this behavior I was running with munmap disabled. We are currently running with it enabled, to see how much it helps or hurts us. Private_Clean is still high with munmap enabled, however our processes RSS seems to be growing slower. Also, we have swapping disabled. Mainly because we are running a bunch of java processes for which we don't want any of the JVM to ever be swapped. It's unclear if enabling swap will help us avoid the OS OOM killer, however we will soon be running some experiments with swap enabled. I've been try to catch strace when the Private_Clean is growing. Not sure if you are asking for a full strace or whether the summary is good enough. A full strace would be way too large as our server is under full load with tens of thousands of allocations every second. The summary is attached below for a 30 minute duration, during which time I observed a single smap grow by 100MB of private clean pages. Over the next couple of hours it didn't grow at all, rather it shrunk by a couple of hundred MBs. Please let me know if anything else is needed, or if you have any suggestions with other jemalloc config options or Linux tuning we maybe able to do to help. THX ! 1) Linux version: Linux 10.42.229.68 2.6.32-131.21.1.89.br5_0.x86_64 #1 SMP Tue Mar 19 15:05:57 CDT 2013 x86_64 GNU/Linux 2) config options for jemalloc: =============================================================================== jemalloc version : 3.3.1-0-g9ef9d9e8c271cdf14f664b871a8f98c827714784 library revision : 1 CC : gcc CPPFLAGS : -D_GNU_SOURCE -D_REENTRANT CFLAGS : -std=gnu99 -Wall -pipe -g3 -fvisibility=hidden -O3 -funroll-loops LDFLAGS : LIBS : -lm -lpthread RPATH_EXTRA : XSLTPROC : /usr/bin/xsltproc XSLROOT : /usr/share/sgml/docbook/xsl-stylesheets PREFIX : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc BINDIR : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc/bin INCLUDEDIR : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc/include LIBDIR : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc/lib DATADIR : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc/share MANDIR : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc/share/man srcroot : abs_srcroot : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc/src/jemalloc-3.3.1/ objroot : abs_objroot : /root/kWebSphere/workspace/XS3/XS/ws/code/prereq.jemalloc/src/jemalloc-3.3.1/ JEMALLOC_PREFIX : je JEMALLOC_PRIVATE_NAMESPACE : install_suffix : autogen : 0 experimental : 1 cc-silence : 0 debug : 0 stats : 1 prof : 0 prof-libunwind : 0 prof-libgcc : 0 prof-gcc : 0 tcache : 0 fill : 1 utrace : 0 valgrind : 0 xmalloc : 0 mremap : 0 munmap : 1 dss : 0 lazy_lock : 0 tls : 1 =============================================================================== 3) jemalloc runtime: Version: 3.3.1-0-g9ef9d9e8c271cdf14f664b871a8f98c827714784 Assertions disabled Run-time option settings: opt.abort: false opt.lg_chunk: 21 opt.dss: "secondary" opt.narenas: 96 opt.lg_dirty_mult: 6 opt.stats_print: false opt.junk: false opt.quarantine: 0 opt.redzone: false opt.zero: false CPUs: 24 Arenas: 96 Pointer size: 8 Quantum size: 16 Page size: 4096 Min active:dirty page ratio per arena: 64:1 Chunk size: 2097152 (2^21) 4) strace summary bye, kMartin . <table width=100% style="border-collapse:collapse;"> <tr valign=top height=8> <td width=96 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">From: <td style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">Jason Evans <jasone@canonware.com> <tr valign=top height=8> <td width=96 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">To: <td style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">Kurtis Martin/Raleigh/IBM@IBMUS, <tr height=8> <td width=96 valign=top style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">Cc: <td style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">jemalloc-discuss@canonware.com <tr valign=top height=8> <td width=96 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">Date: <td style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">06/03/2013 07:23 PM <tr valign=top height=8> <td width=96 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">Subject: <td style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">Re: High amount of private clean data in smaps</table> <hr noshade> On May 30, 2013, at 4:06 PM, Kurtis Martin wrote: 1) Why does jemalloc have smaps with such large Private_Clean size? I'm actually surprised jemalloc has such large smaps in general. I would expect a bunch of smaller smaps that match the configured chunk size. I've been trying to figure this out for quite a while now, and I have yet to come up a way to transition pages that were mapped as MAP_PRIVATE|MAP_ANON to the Private_Clean state. My experiments included fork(2) abuse, mmap'ed files, shared anonymous memory, etc., and I'm currently out of ideas. If you're able to observe a process as its Private_Clean page count is increasing, can you capture an strace log to see what system calls are occurring? Also, can you tell me the Linux kernel version you're using, jemalloc configuration (e.g. whether munmap is disabled), and jemalloc run-time options specified? Regarding large smaps, all the Unix operating systems I've dealt with coalesce mappings that have identical attributes. If jemalloc maps two chunks that happen to be adjacent to each other, the kernel tracks them as a single mapping. jemalloc goes to some effort to make coalescing possible, because Linux unfortunately does linear map scans that severely degrade performance if the number of map entries isn't kept low. Thanks, Jason