Memory usage regression

[Mike Hommey]

On Tue, Oct 30, 2012 at 07:01:21PM +0100, Mike Hommey wrote:
> On Tue, Oct 30, 2012 at 05:03:38PM +0100, Mike Hommey wrote:
> > So, what seems to be happening is that because of that fragmentation, when
> > requesting big allocations, jemalloc has to allocate and use new chunks.
> > When these big allocations are freed, the new chunk tends to be used
> > more often than the other free chunks, adding to the fragmentation, thus
> > requiring more new chunks for big allocations.
> > 
> > The following dumb patch essentially plugs the leak for the Firefox usecase:
> > 
> > diff --git a/src/arena.c b/src/arena.c
> > index 1e6964a..38079d7 100644
> > --- a/src/arena.c
> > +++ b/src/arena.c
> > @@ -471,7 +471,7 @@ arena_run_alloc_helper(arena_t *arena, size_t size, bool large, size_t binind,
> >         arena_chunk_map_t *mapelm, key;
> >  
> >         key.bits = size | CHUNK_MAP_KEY;
> > -       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_dirty, &key);
> > +       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_clean, &key);
> >         if (mapelm != NULL) {
> >                 arena_chunk_t *run_chunk = CHUNK_ADDR2BASE(mapelm);
> >                 size_t pageind = (((uintptr_t)mapelm -
> > @@ -483,7 +483,7 @@ arena_run_alloc_helper(arena_t *arena, size_t size, bool large, size_t binind,
> >                 arena_run_split(arena, run, size, large, binind, zero);
> >                 return (run);
> >         }
> > -       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_clean, &key);
> > +       mapelm = arena_avail_tree_nsearch(&arena->runs_avail_dirty, &key);
> >         if (mapelm != NULL) {
> >                 arena_chunk_t *run_chunk = CHUNK_ADDR2BASE(mapelm);
> >                 size_t pageind = (((uintptr_t)mapelm -
> > 
> > 
> > My test program changed in the meanwhile, so i can't do accurate
> > comparisons with mozjemalloc without re-running more tests. I'll post
> > again when I have more data.

Oops. There was supposed to be an url around here:
http://i.imgur.com/A0len.png

> Here's some comparison between jemalloc3 (tcache=false, narenas=1,
> lg_chunk=20) with the patch above and mozjemalloc.
> The graph shows (mozjemalloc - jemalloc3) / jemalloc3 for each value.
> 
> mozjemalloc has a lower rss after closing tabs, but jemalloc3 has a
> lower rss when all tabs are opened. VmData difference is in an
> acceptable range.
> "Usable" is the total of malloc_usable_size() for all allocations, which
> means jemalloc allocates more "unrequested" memory than mozjemalloc.
> That could certainly contribute to some additional fragmentation and
> to the RSS difference, by extension.
> 
> Mike