<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Jeff,<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Agree, this is a well know problem. It does not work well with some applications.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We will provide this option to our users and they will have the freedom to enable/disable this.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best,<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Pasha<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Jeff Hammond [mailto:jeff.science@gmail.com] <br> <b>Sent:</b> Tuesday, October 20, 2015 5:54 PM<br> <b>To:</b> Shamis, Pavel<br> <b>Cc:</b> jemalloc-discuss@canonware.com; Baker, Matthew B.; Yossi Etigin<br> <b>Subject:</b> Re: Memory allocation/release hooks<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal">You don't want to use malloc interception if you care about portability/interoperability. For example, see <a href="http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2014-December/005276.html"> http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2014-December/005276.html</a>.<o:p></o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">I can't find the archives of <a href="mailto:madness-developers@googlegroups.com">madness-developers@googlegroups.com</a> online (perhaps I am just stupid) but we recently encountered the same issue with JEMalloc and other MPI implementations.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">I understand why it is tempting to intercepting malloc/free for MPI purposes, but the end result is brittle software that forces users to disable all the optimizations you are trying to enable.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">And it is worth noting that the abuse of malloc/free interception by MPI developers has forced the MADNESS team (or rather me) to completely bypass ISO C/C++ language defined heap routines to prevent MPI from hijacking them and breaking our code.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Anyways, this is in no way a statement about JEMalloc. The results with MADNESS indicate this it is the best available allocator around (vs GNU, TCE and TBB mallocs), but we will have to call it explicitly rather than via symbol interception.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><br> Best,<o:p></o:p></p> </div> <div> <p class="MsoNormal"><br> Jeff<o:p></o:p></p> </div> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal">On Tue, Oct 20, 2015 at 12:18 PM, Shamis, Pavel <<a href="mailto:shamisp@ornl.gov" target="_blank">shamisp@ornl.gov</a>> wrote:<o:p></o:p></p> <div> <p class="MsoNormal">Hi Jeff, <o:p></o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Thanks for the link, seems like a very useful library. <o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Our goal is a bit different (and very simple/basic). <o:p></o:p></p> </div> <div> <p class="MsoNormal">We are looking for a malloc library that we can use for integration with our registration cache.<o:p></o:p></p> </div> <div> <p class="MsoNormal">Essentially, it redirects application's malloc() calls to (through LD_PRELOAD or rpath) jemalloc that is hooked up with a cache (just like in HPX).<o:p></o:p></p> </div> <div> <p class="MsoNormal">At this stage we don't play with locality. <o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Thanks !<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <div> <div> <p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black">Pavel (Pasha) Shamis<o:p></o:p></span></p> </div> <div> <p class="MsoNormal" style="margin-bottom:13.5pt"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black">---<br> Computer Science Research Group<br> Computer Science and Math Division<br> Oak Ridge National Laboratory<br> <br> <br> <br> <br> <o:p></o:p></span></p> </div> </div> </div> </div> <div> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <p class="MsoNormal">On Oct 20, 2015, at 11:31 AM, Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>> wrote:<o:p></o:p></p> </div> <p class="MsoNormal"><br> <br> <o:p></o:p></p> <div> <p class="MsoNormal">Hi Pavel, <o:p></o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">You may find <a href="http://memkind.github.io/memkind/" target="_blank">http://memkind.github.io/memkind/</a> relevant. In particular, <a href="http://memkind.github.io/memkind/memkind_arch_20150318.pdf" target="_blank">http://memkind.github.io/memkind/memkind_arch_20150318.pdf</a> section 2.2 and 2.3 discusses exactly the issues you raise. We also note that memkind is intended to support multiple types of memory within a node, such as one might encounter in a platform such as Knights Landing. You are free to imagine how it might map to OpenPOWER based upon your superior knowledge of that platform :-)<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">While I recognize that the origins of memkind at Intel may pose a challenge for some in the OpenPOWER family, it would be tremendously valuable to the community if it was reused for MPI and OpenSHMEM projects, rather than the UCX team trying to implement something new. As you know, the both MPI and OpenSHMEM should run on a range of platforms, and it doubles the implementation effort in all relevant projects (MPICH, OpenMPI, OpenSHMEM reference, etc.) if UCX goes in a different direction.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">I would be happy to introduce you to the memkind developers (I am not one of them, just someone who helps them understand user/client requirements).<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Best,<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Jeff<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal">On Thu, Oct 15, 2015 at 8:45 AM, Shamis, Pavel <<a href="mailto:shamisp@ornl.gov" target="_blank">shamisp@ornl.gov</a>> wrote:<o:p></o:p></p> <p class="MsoNormal">Dear Jemalloc Community,<br> <br> We are developer of UCX project [1] and as part of the effort<br> we are looking for a malloc library that supports hooks for alloc/dealloc chunks and can be used for the following:<br> <br> (a) Allocation of memory that can be shared transparently between processes on the same node. For this purpose we would like to mmap memory with MAP_SHARED. This is very useful for implementation for Remote Memory Access (RMA) operations in MPI-3 one-sided [2] and OpenSHMEM [3] communication libraries. This allow a remote process to map user allocated memory and provide RMA operations through memcpy().<br> <br> (b) Implementation of memory de-allocation hooks for RDMA hardware (Infiniband, ROCE, iWarp etc.). For optimization purpose we implement a lazy memory de-registration (memory unpinning) policy and we use the hook for the notification of communication library about memory release event. On the event, we cleanup our registration cache and de-register (unpin) the memory on hardware.<br> <br> Based on this requirements we would like to understand what is the best approach for integration this functionality within jemalloc.<br> <br> Regards,<br> Pasha & Yossi<br> <br> [1] OpenUCX: <a href="https://github.com/openucx/ucx" target="_blank">https://github.com/openucx/ucx</a> or <a href="http://www.openucx.org/" target="_blank">www.openucx.org</a><br> [2] MPI SPEC: <a href="http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf" target="_blank"> http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf</a><br> [3] OpenSHMEM SPEC: <a href="http://bongo.cs.uh.edu/site/sites/default/site_files/openshmem-specification-1.2.pdf" target="_blank"> http://bongo.cs.uh.edu/site/sites/default/site_files/openshmem-specification-1.2.pdf</a><br> <br> <br> <br> <br> <br> _______________________________________________<br> jemalloc-discuss mailing list<br> <a href="mailto:jemalloc-discuss@canonware.com" target="_blank">jemalloc-discuss@canonware.com</a><br> <a href="http://www.canonware.com/mailman/listinfo/jemalloc-discuss" target="_blank">http://www.canonware.com/mailman/listinfo/jemalloc-discuss</a><o:p></o:p></p> </div> <p class="MsoNormal"><br> <br clear="all"> <o:p></o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <p class="MsoNormal">-- <o:p></o:p></p> <div> <p class="MsoNormal">Jeff Hammond<br> <a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br> <a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a><o:p></o:p></p> </div> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> </div> </div> </div> <p class="MsoNormal"><br> <br clear="all"> <o:p></o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <p class="MsoNormal">-- <o:p></o:p></p> <div> <p class="MsoNormal">Jeff Hammond<br> <a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br> <a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a><o:p></o:p></p> </div> </div> </div> </div> </body> </html>