allocating memory above 2GB

[Jason Evans]

On Feb 14, 2014, at 1:52 PM, Evan Wies <evan at neomantra.net> wrote:
> Is it possible to allocate memory in a defined region, specifically anywhere above the first 2GB?
> 
> I see that mallocx with ALLOCM_ALIGN allows one to specify a base address aligned to a power of 2.  Can I grab a bunch of memory that way and somehow tell an arena to use that space and then always mallocx with that arena?
> 
> I'm in the midst of writing a LuaJIT [1] binding to jemalloc, and have successfully allocated memory aligned to 2^31 but can't figure out the next step -- short of writing my own allocator to dole out jemalloc-allocated memory.
> 
> Context:
> LuaJIT, a Lua Just-In-Time compiler, can itself only allocate memory in the lower portions of memory (<1GB, <2GB, <4GB depending on platform).   One can get around this limitation by using its FFI to allocate memory for FFI-bound C structures, however allocating these in the lower regions effectively "steals" available memory from the LuaJIT VM for Lua objects.   I want to ensure maximum available memory for the LuaJIT by allocating all my objects outside this area.

By default jemalloc prefers mmap() over sbrk() when allocating memory, which for 64-bit systems means that it won't allocate the low memory LuaJIT wants until mmap() fails.  On 32-bit systems, I think there are non-trivial differences between operating systems in terms of where mmap() will place anonymous mappings; IIRC FreeBSD uses a fixed (but tunable) amount of memory for sbrk(), whereas Linux allows sbrk() and mmap() to collide (but I don't remember where code gets mapped, so the practical result may be similar to FreeBSD).  I suspect that jemalloc's preference for mmap() will be sufficient for all but the most memory-hungry Lua applications, even on 32-bit systems.  Take a look at where jemalloc is actually allocating memory from; it's probably above where code is mapped.

If you want to get hacky, you can build jemalloc with munmap() usage disabled, temporarily sbrk() all the low memory LuaJIT could possibly want, then temporarily allocate all the malloc()ed memory you could possibly want, then release both.  The result will be that jemalloc has a pool of cached chunks that can't have come from low memory.

Jason