<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> <p class="MsoNormal"><span style="color:#1F497D">Turns out transparent_hugepages is set to “always” on 14.04, was set to “madvise” on 12.04. Should it be set to “never” everywhere rather than “madvise”?<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D">Testing now… will update here when results are clear.<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D">Sheppard<o:p></o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Jason Evans [mailto:jasone@canonware.com] <br> <b>Sent:</b> Tuesday, June 03, 2014 2:36 PM<br> <b>To:</b> Sheppard Parker<br> <b>Cc:</b> jemalloc-discuss@canonware.com<br> <b>Subject:</b> Re: RES memory footprint on Ubuntu 14.04<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">On Jun 3, 2014, at 11:41 AM, Sheppard Parker <<a href="mailto:Sheppard.Parker@maxpoint.com">Sheppard.Parker@maxpoint.com</a>> wrote:<o:p></o:p></p> <div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal">We have been running a particular jemalloc-aware application on Ubuntu Server 12.04 LTS for about a year now with no trouble (thx btw… we were having major fragmentation issues prior to trying jemalloc). Nominal RES memory footprint is between 23GB and 27GB depending on how much data is loaded at any given time of the day or day of the week.<o:p></o:p></p> <p class="MsoNormal"> <o:p></o:p></p> <p class="MsoNormal">We recently purchased some new hardware that came with Ubuntu Server 14.04 LTS and now our same app shows almost 2X the RES memory footprint when loaded with the same data. I don’t think it is growing, so it may not be a big deal, but I am still concerned. Why the major difference? Exact same executable installed, no code changes, even the same hardware (Dell R620s). Only difference is the move from Ubuntu Server 12.04 (3.8.0-36-generic #52~precise1-Ubuntu SMP) to Ubuntu Server 14.04 (one machine is 3.13.0-24-generic #47-Ubuntu SMP, another is 3.13.0-27-generic #50-Ubuntu SMP).<o:p></o:p></p> <p class="MsoNormal"> <o:p></o:p></p> <p class="MsoNormal">Any ideas? Is there something enabled/disabled on Ubuntu 14.04 that I need to modify to get things working like they do on the older 12.04? FWWIW, we have been using jemalloc v3.5.0.<o:p></o:p></p> <p class="MsoNormal"> <o:p></o:p></p> <p class="MsoNormal">I tried rebuilding our app (again, no code changes) with jemalloc 3.5.1 and 3.6.0. No change. Seems like jemalloc is unhappy about something on Ubuntu 14.04 (or visa versa), but what? Has anybody else encountered anything similar? <o:p></o:p></p> </div> </blockquote> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">My only guess is that the newer system is using transparent huge pages, whereas the older one is not. I recently came across this related blog post:<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span class="apple-tab-span"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""> </span></span><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><a href="http://dev.nuodb.com/techblog/linux-transparent-huge-pages-jemalloc-and-nuodb">http://dev.nuodb.com/techblog/linux-transparent-huge-pages-jemalloc-and-nuodb</a><o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">In short, madvise() isn't actually purging unused dirty (4 KiB) pages if the underlying memory has been promoted to huge (2 MiB) pages. In the short term, the solution seems to be disabling transparent huge pages. In the long run I hope the Linux kernel improves its algorithms for such usage patterns, and I'm also contemplating layout strategies that coexist better with huge pages.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">Jason<o:p></o:p></span></p> </div> </div> </body> </html>