Allocation latency during fork

[Jason Evans]

On Jul 4, 2014, at 2:02 AM, Salvatore Sanfilippo <antirez at gmail.com> wrote:
> while trying to profile my application for sources of latency, I
> noticed that after the fork() call, if there are an high number of
> allocations ongoing, one of the next allocations (the first maybe?)
> after fork()  shows very high latency (~400 milliseconds) in a process
> using 2GB of memory.
> 
> The problem exists both in jemalloc 3.2.0 and 3.6.0 so it does not
> look like a regression.
> I tried with the glibc standard allocator and I can't reproduce the
> issue, so looks like allocator-specific.
> 
> There is some way I can mitigate ore remove this issue?

I'm guessing that jemalloc is accessing a lot of pages (dirty page purging?), and copy-on-write overhead due to the fork is hurting a lot.    The best way I know of to avoid this overhead is to "pre-fork" one or more processes early during application startup (before memory usage grows large), and communicate with these via pipes to fork+exec processes.  This avoids marking the 2 GB of pages in your main process as copy-on-write during fork, which reduces fork overhead as well as page fault rate.  You can take a look at hhvm's LightProcess class for an example implementation:

	https://github.com/facebook/hhvm/blob/master/hphp/util/light-process.h
	https://github.com/facebook/hhvm/blob/master/hphp/util/light-process.cpp

Jason