Memory allocation/release hooks

[Shamis, Pavel]

>>>>
>>>>(b) Implementation of memory de-allocation hooks for RDMA hardware
>>>>(Infiniband, ROCE, iWarp etc.). For optimization purpose we implement
>>>>a lazy memory de-registration (memory unpinning) policy and we use the
>>>>hook for the  notification of communication library about memory
>>>>release event. On the event, we cleanup our registration cache and
>>>>de-register (unpin) the memory on hardware.
>>> 
>>> We have been using jemalloc for some time to manage, among other
>>>things, registered memory regions in HPX-5 (https://hpx.crest.iu.edu/)
>>>for Verbs and uGNI. If you already have a mechanism which manages keys,
>>>then you can simply install a set of chunk hooks that can perform the
>>>registration/deregistration as necessary. We have found this to work
>>>quite well for our purposes.
>> 
>> How do you load jemalloc ? Do you do LD_PRELOAD  or the user is expect
>>to allocate the memory explicitly through HPX runtime ?
>
>We have allocator choice set up as a configuration-time dependency so
>it¹s relatively straightforward.
>
>When the user has selected jemalloc as the allocator (it¹s our default
>but we have a couple of other options) then a libjemalloc.so that
>supports the version 4 API is a configure-time requirement. We will just
>-rpath it into libhpx.so, and/or export it through hpx.pc for libhpx.a.

Got it !

>
>PS: As an aside, we have [this outstanding
>issue](https://github.com/jemalloc/jemalloc/issues/203) which would be
>useful to us for key management. Since jemalloc already stores chunk
>metadata we¹re hoping that we could get 16-64 bytes of ³user² metadata
>per chunk so we don¹t need to keep track of our own keys. Hopefully this
>will become available at some point.

It is a very convenient feature to have. Our plan was to implement own
caching mechanism.
Side note. By default you don't really want to invoke registration for
every user memory allocation (this would be too expensive in term of
memory consumption). Typically, we register the memory on a first RMA
access. 

Regards,
Pasha