Memory allocation/release hooks

[D'Alessandro, Luke K]

> On Oct 15, 2015, at 2:26 PM, Shamis, Pavel <shamisp at ornl.gov> wrote:
> 
> [cut]
> 
>>> (b) Implementation of memory de-allocation hooks for RDMA hardware (Infiniband, ROCE, iWarp etc.). For optimization purpose we implement a lazy memory de-registration (memory unpinning) policy and we use the hook for the  notification of communication library about memory release event. On the event, we cleanup our registration cache and de-register (unpin) the memory on hardware.
>> 
>> We have been using jemalloc for some time to manage, among other things, registered memory regions in HPX-5 (https://hpx.crest.iu.edu/) for Verbs and uGNI. If you already have a mechanism which manages keys, then you can simply install a set of chunk hooks that can perform the registration/deregistration as necessary. We have found this to work quite well for our purposes.
> 
> How do you load jemalloc ? Do you do LD_PRELOAD  or the user is expect to allocate the memory explicitly through HPX runtime ?

We have allocator choice set up as a configuration-time dependency so it’s relatively straightforward. 

When the user has selected jemalloc as the allocator (it’s our default but we have a couple of other options) then a libjemalloc.so that supports the version 4 API is a configure-time requirement. We will just -rpath it into libhpx.so, and/or export it through hpx.pc for libhpx.a.

We have made a decision to distribute all of our default dependencies, so we will actually build a libjemalloc.so from source and install that if we can’t find one at configuration time. We have made the decision to use it without a prefix, which means it will also be used for local allocation, but it mostly performs better than the libc malloc that’s installed on the systems that we use so that doesn’t concern us. A more complicated configuration could support prefixing jemalloc and leaving the local allocation alone.

Luke

PS: As an aside, we have [this outstanding issue](https://github.com/jemalloc/jemalloc/issues/203) which would be useful to us for key management. Since jemalloc already stores chunk metadata we’re hoping that we could get 16-64 bytes of “user” metadata per chunk so we don’t need to keep track of our own keys. Hopefully this will become available at some point.