Allocation latency during fork

[Salvatore Sanfilippo]

Hello, finally I discovered the cause of the issue, so I want to follow up.

For some reason I did not investigated, libc malloc allocates memory
in a way does not allow transparent huge pages to work for that
region.
Jemalloc allocated memory instead is targeted by transparent huge
pages, so when the feature is enabled, after fork, we have processes
sharing the same huge pages.

Once one of the processes starts writing to many pages in a very fast
way, multiple page faults occur, triggering the copy-on-write of
almost all the process space (It takes a few thousands writes to COW a
2GB process, for example). Since huge pages copying is slow, thousands
of copy operations sum up to hundreds of milliseconds.

Disabling transparent huge pages makes the problem go away.

>From the point of view of jemalloc, would be cool to have some option
in order to mmap memory so that it is not targeted by THP regardless
of the kernel global settings.

Thanks,
Salvatore

On Fri, Jul 4, 2014 at 11:02 AM, Salvatore Sanfilippo <antirez at gmail.com> wrote:
> Hello,
>
> while trying to profile my application for sources of latency, I
> noticed that after the fork() call, if there are an high number of
> allocations ongoing, one of the next allocations (the first maybe?)
> after fork()  shows very high latency (~400 milliseconds) in a process
> using 2GB of memory.
>
> The problem exists both in jemalloc 3.2.0 and 3.6.0 so it does not
> look like a regression.
> I tried with the glibc standard allocator and I can't reproduce the
> issue, so looks like allocator-specific.
>
> There is some way I can mitigate ore remove this issue?
>
> Thanks,
> Salvatore
>
> --
> Salvatore 'antirez' Sanfilippo
> open source developer - GoPivotal
> http://invece.org
>
> "One would never undertake such a thing if one were not driven on by
> some demon whom one can neither resist nor understand."
>        — George Orwell



-- 
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

"Fear makes the wolf bigger than he is."
       — German proverb