Managing pinned memory

[Jason Evans]

On May 8, 2014, at 9:00 AM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
> I’m in the market for a good concurrent allocator to manage a memory region corresponding to pinned network memory for a multithreaded and distributed HPC application. Basically, I’m going to want to do RDMA to objects that are often malloced and freed. The pinning operation is expensive so it is important to amortize it over lots of uses. I’ve written a simple thread-local caching allocator that allows me to pin contiguous blocks when they’re first allocated, and then just use TLS free listing to reuse space, however I don’t really have the resources needed to implement this in a robust way.
> 
> Is there any natural way to do this in jemalloc at this time? My gut feeling is that there isn’t, explicitly specifying an arena breaks its caching and there’s not an obvious way to register a callback to run on internal block allocation and freeing (where I could pin/unpin the underlying memory).
> 
> If jemalloc doesn’t really support this use case, does anyone know of an efficient, scalable, robust allocator that does?

This pending change may be relevant to your needs:

	https://github.com/jemalloc/jemalloc/pull/80

I’m imagining that you would implement a custom chunk allocator that pins entire chunks, and then specifically use that arena for allocations that you require to be pinned.  This approach has some shortcomings, but perhaps they don’t matter to your specific application.

Jason