Excessive VM usage with jemalloc

[Jason Evans]

On May 6, 2013, at 6:46 AM, Abhishek Singh wrote:
> On Sat, May 4, 2013 at 10:14 PM, Jason Evans <jasone at canonware.com> wrote:
> On Apr 28, 2013, at 9:03 PM, Abhishek Singh <abhishek at abhishek-singh.com> wrote:
> > Our setups start typically with 96 GB of RAM and up. We have observed that using jemalloc the virtual memory usage of our process rises up to around 75GB. While the resident memory stays low and it is not a problem as such, when we try to fork a process from within here, it fails as the kernel assumes there is not enough memory to copy the VM space. Perhaps a vfork would be better but we can't use that for now.
> >
> > So we have made some modifications to jemalloc so that all huge memory allocations are forced to unmap their memory on freeing up. Non huge memory allocations and freeing up remain the same. This seems to help us. I have attached the patch here which is against jemalloc-3.3.1. Please review and suggest if there is a better way to handle this.
> 
> Does --enable-munmap give you similar results?  Ideally, munmap() would always be enabled, but Linux has some unfortunate VM map fragmentation issues (it doesn't consistently reuse holes left by munmap()).  I don't think there's much benefit to using munmap() only selectively, because avoiding the VM map fragmentation issue is an all-or-nothing proposition.
> 
> Yes, results with --enable-munmap are similar but some preliminary benchmarks for the kind of allocations we do, seem to indicate degradation in performance as compared to unmap disabled. While you are right and ideally it would be great to unmap all the memory as and when it is not needed, if there is a performance tradeoff then it may not be feasible. Our aim as of now is to ensure that we cause minimal memory fragmentation (thereby reducing our virtual memory footprint) and still get as good a performance as we can. Selectively unmapping only huge allocations seemed to be a compromise between the two extremes.

Does your application use fork() just to call exec()?  If so, you can pre-fork one or more helper processes (and communicate with them via pipes) early during execution that act as fork()/exec() helpers.  By creating them early on, you assure that they have a much smaller memory footprint than the huge main process, which will improve both speed and reliability.

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jemalloc.net/mailman/jemalloc-discuss/attachments/20130506/6f09d214/attachment.html>