<html><body> With help from our local Linux kernel experts we've tracked down the inexplicable Private_Clean emergence in our processes' smaps file to a kernel bug, <a href="https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/proc/task_mmu.c?id=1c2499ae87f828eabddf6483b0dfc11da1100c07">https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/proc/task_mmu.c?id=1c2499ae87f828eabddf6483b0dfc11da1100c07</a>, which, according to GIT, was first committed in v2.6.36-rc6~63. When we manually applied the aforementioned patched to our kernel there were no memory segments in smaps showing large Private_Clean regions during our test. Unfortunately the fix seems to have been merely an accounting change. Everything previously reported as Private_Clean is now correctly showing up as Private_Dirty, so we are still digging to find out why our RSS, specifically Private_Dirty, continues to grow while jemalloc's active reports much lower numbers. <ul style="padding-left: 0pt"> Thanks, Tom</ul> <img width="16" height="16" src="cid:1__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for Jason Evans ---06/06/2013 01:34:42 AM---On Jun 5, 2013, at 9:17 PM, Thomas R Gissel < gissel@us.ibm.c">Jason Evans ---06/06/2013 01:34:42 AM---On Jun 5, 2013, at 9:17 PM, Thomas R Gissel < gissel@us.ibm.com <table width="100%" border="0" cellspacing="0" cellpadding="0"> <tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> <ul style="padding-left: 4pt">From:</ul> </td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> Jason Evans <jasone@canonware.com></td></tr> <tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> <ul style="padding-left: 4pt">To:</ul> </td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> Thomas R Gissel/Rochester/IBM@IBMUS, </td></tr> <tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> <ul style="padding-left: 4pt">Cc:</ul> </td><td width="100%" valign="middle"><img width="1" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> jemalloc-discuss@canonware.com</td></tr> <tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> <ul style="padding-left: 4pt">Date:</ul> </td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> 06/06/2013 01:34 AM</td></tr> <tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> <ul style="padding-left: 4pt">Subject:</ul> </td><td width="100%"><img width="1" height="1" src="cid:2__=08BBF106DFDBB6238f9e8a93df938@us.ibm.com" border="0" alt=""> Re: High amount of private clean data in smaps</td></tr> </table> <hr width="100%" size="2" align="left" noshade style="color:#8091A5; "> On Jun 5, 2013, at 9:17 PM, Thomas R Gissel <<a href="mailto:gissel@us.ibm.com">gissel@us.ibm.com</a>> wrote: <ul style="padding-left: 36pt"> I too have been trying to reproduce the existence of Private_Clean memory segments in smaps via a simple test case with jemalloc and was unable to on my laptop, a 2 core machine running a 3.8.0-23 kernel . I then moved my test to our production box: 96GB memory, 24 hardware threads and 2.6 kernel (detailed information below), and within a few minutes of execution, with a few minor adjustments, I was able duplicate the results, smaps showing the jemalloc segment with Private_Clean memory usage, of our larger test. Note that I'm using the same jemalloc library whose information Kurtis posted earlier (96 arenas etc...). </ul> Interesting! I don't see anything unusual about the test program, so I'm guessing this is kernel-specific. I'll run it on some 8- and 16-core machines tomorrow with a couple of kernel versions and see what happens. Thanks, Jason </body></html>