Memory usage regression

[Mike Hommey]

On Tue, Oct 30, 2012 at 04:36:58PM +0100, Mike Hommey wrote:
> On Tue, Oct 30, 2012 at 04:35:02PM +0100, Mike Hommey wrote:
> > On Fri, Oct 26, 2012 at 06:10:13PM +0200, Mike Hommey wrote:
> > > > > > For reference, the unzoomed graph looks like this:
> > > > > > http://i.imgur.com/PViYm.png
> > > > > 
> > > > > I rediscovered --enable-munmap, and tried again with that, thinking it
> > > > > could be related, and it did change something, but it's still growing:
> > > > > http://i.imgur.com/lWzhG.png
> > > > 
> > > > Needless to say, the increases I was observing closely on the the zoomed
> > > > graph without a matching decrease was entirely due to munmap. Now I need
> > > > to find the remainder...
> > > 
> > > I tested size class independently, and none would cause the VM leak
> > > alone. Combining small and large classes do, but large + huge or small +
> > > huge don't.
> > 
> > Some more data: all non-unmapped chunks *are* used to some extent. The
> > following is a dump of the number of requested and usable bytes in each
> > chunk ; that's 18M spread across 600M... that sounds like a really bad
> > case of fragmentation.
> 
> BTW, it does seem to grow forever: I went up to 1.3GB with more
> iterations before stopping.

So, what seems to be happening is that because of that fragmentation, when
requesting big allocations, jemalloc has to allocate and use new chunks.
When these big allocations are freed, the new chunk tends to be used
more often than the other free chunks, adding to the fragmentation, thus
requiring more new chunks for big allocations.

The following dumb patch essentially plugs the leak for the Firefox usecase:

diff --git a/src/arena.c b/src/arena.c
index 1e6964a..38079d7 100644
--- a/src/arena.c
+++ b/src/arena.c
@@ -471,7 +471,7 @@ arena_run_alloc_helper(arena_t *arena, size_t size, bool large, size_t binind,
        arena_chunk_map_t *mapelm, key;
 
        key.bits = size | CHUNK_MAP_KEY;
-       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_dirty, &key);
+       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_clean, &key);
        if (mapelm != NULL) {
                arena_chunk_t *run_chunk = CHUNK_ADDR2BASE(mapelm);
                size_t pageind = (((uintptr_t)mapelm -
@@ -483,7 +483,7 @@ arena_run_alloc_helper(arena_t *arena, size_t size, bool large, size_t binind,
                arena_run_split(arena, run, size, large, binind, zero);
                return (run);
        }
-       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_clean, &key);
+       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_dirty, &key);
        if (mapelm != NULL) {
                arena_chunk_t *run_chunk = CHUNK_ADDR2BASE(mapelm);
                size_t pageind = (((uintptr_t)mapelm -


My test program changed in the meanwhile, so i can't do accurate
comparisons with mozjemalloc without re-running more tests. I'll post
again when I have more data.

Mike