Regression: jemalloc >= 4.0 use munmap() even when configured with --disable-munmap

[Daniel Mewes]

Hi Jason,

thank you for your reply.

The reason for the failing `munmap` appears to be that we hit the kernel's
`max_map_count` limit.

I can reproduce the issue very quickly by reducing the limit through `echo
16000 > /proc/sys/vm/max_map_count`, and it disappears in our tests when
increasing it to something like `echo 131060 > /proc/sys/vm/max_map_count`.
The default value is 65530 I believe.

We used to see this behavior in jemalloc 2.x, but didn't see it in 3.x
anymore. It now re-appeared somewhere between 3.6 and 4.1.

It looks like I looked at the wrong place when I checked the jemalloc 3.6
code for comparison earlier today, and I can now see that the same code was
indeed there just in a different file (`chunk_mmap.c`). Thanks for
clarifying this.

So it seems that the difference between 3.6 and 4.1 must be caused by
something else then, and we might just have been lucky that the particular
behavior of jemalloc 3 didn't trigger the issue for our workload.

Do you think the allocator should handle reaching the map_count limit and
somehow deal with it gracefully (if that's even possible)? Or should we
just advise our users to raise the kernel limit, or alternatively try to
change RethinkDB's allocation patterns to avoid hitting it?

I can try to come up with a small test case to specifically reproduce this
issue later.

- Daniel



On Fri, Apr 22, 2016 at 9:24 PM, Jason Evans <jasone at canonware.com> wrote:

> On Apr 22, 2016, at 4:38 PM, Daniel Mewes <daniel at rethinkdb.com> wrote:
> > In jemalloc 3.0, this patch added the `--disable-munmap` option and
> disabled the use of `munmap` on Linux by default:
> https://github.com/jemalloc/jemalloc/commit/59ae2766af88bad07ac721c4ee427b171e897bcb
> >
> > It looks like jemalloc starting with version 4.0 makes use of `munmap`
> even when `--disable-munmap` is specified. From what I can tell,
> `chunk_map.c` honors the `config_munmap` flag, but the function
> `page_unmap` in `pages.c` ignores it (this code appears to be new in
> jemalloc 4?).
> >
> > We are using jemalloc for RethinkDB and would like to upgrade to version
> 4.1 because we think that it fixes some bugs that our users have run into.
> > However it causes a regression for
> https://github.com/rethinkdb/rethinkdb/issues/3516 :
> > "<jemalloc>: Error in munmap(): Cannot allocate memory"
>
> pages_unmap() is used to trim mappings so that what remains is
> chunk-aligned, regardless of whether --disable-munmap is specified.
> jemalloc 3.x has similar code that calls munmap().  I don't see anything in
> what you're describing that is particular to jemalloc 4.x.  Are you able to
> determine anything else about the failure?  Its' extremely unusual for
> munmap() to fail (I've not seen it happen since ~2005 during initial
> development), so I'm guessing there's a memory corruption issue of some
> sort, whether due to a bug in jemalloc or RethinkDB.
>
> Thanks,
> Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jemalloc.net/mailman/jemalloc-discuss/attachments/20160422/9fcc0f5c/attachment.html>