mremap with modern Linux kernel

[Daniel Micay]

On 25/04/14 06:21 PM, Jason Evans wrote:
> On Apr 25, 2014, at 2:55 AM, Daniel Micay <danielmicay at gmail.com> wrote:
>> This option was originally disabled by default due to fragmentation
>> issues. It provides a significant performance win for Rust's vectors at
>> very large sizes, so I'm curious about the severity of this issue and
>> whether it is still around in the latest Linux kernel releases.
> 
> As far as I know, this problem still exists in Linux.  The problem is that Linux doesn't have a reliable way to find the first fit for an mmap() request other than linear scan, so it uses heuristics to decide where to start the scan.  It's quite easy to trigger pathological behavior where a chunk of memory is unmapped, but the kernel doesn't revise its scan start point, and the VM map hole remains indefinitely.  The more holes there are, the more mapped regions there are to linearly scan.  I don't remember what the common triggers of linear scans are, but they definitely happen enough to cause a performance issue, at least for some of the heavily loaded network server applications Facebook runs.
> 
> One way to reduce the impact of huge reallocs would be to use exponential size class increases, rather than linear increases.  jemalloc will always round up to the nearest multiple of the chunk size, but it it were to instead use e.g. [4, 8, 16, 32, 64, ...] MiB as size classes, the realloc overhead would amortize away.  I've been thinking about exploring this strategy for large size classes, [4 KiB .. 4 MiB), and I just wrote up a tracking issue that also keeps your use case in mind: 
> 
> 	https://github.com/jemalloc/jemalloc/issues/77
> 
> Thanks,
> Jason

Luckily this isn't a huge issue since the allocation costs are already
amortized out by the container types. I'll have a closer look at the
folly vector type, since it's integrated well with jemalloc.

If I'm understanding correctly the issue is caused by unmapping, so
perhaps jemalloc could still use MREMAP_FIXED and then fall back to
copying. I don't know if it would succeed often enough to be worth it
though.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://jemalloc.net/mailman/jemalloc-discuss/attachments/20140425/49fc37ff/attachment.sig>