Allocation latency during fork
jasone at canonware.com
Tue Jul 8 10:27:55 PDT 2014
On Jul 4, 2014, at 2:02 AM, Salvatore Sanfilippo <antirez at gmail.com> wrote:
> while trying to profile my application for sources of latency, I
> noticed that after the fork() call, if there are an high number of
> allocations ongoing, one of the next allocations (the first maybe?)
> after fork() shows very high latency (~400 milliseconds) in a process
> using 2GB of memory.
> The problem exists both in jemalloc 3.2.0 and 3.6.0 so it does not
> look like a regression.
> I tried with the glibc standard allocator and I can't reproduce the
> issue, so looks like allocator-specific.
> There is some way I can mitigate ore remove this issue?
I'm guessing that jemalloc is accessing a lot of pages (dirty page purging?), and copy-on-write overhead due to the fork is hurting a lot. The best way I know of to avoid this overhead is to "pre-fork" one or more processes early during application startup (before memory usage grows large), and communicate with these via pipes to fork+exec processes. This avoids marking the 2 GB of pages in your main process as copy-on-write during fork, which reduces fork overhead as well as page fault rate. You can take a look at hhvm's LightProcess class for an example implementation:
More information about the jemalloc-discuss