Fragmentation with jemalloc

[vandana shah]

Hi,

I have been trying to use jemalloc for my application and observed that the
rss of the process keeps on increasing.

I ran the application with valgrind to confirm that there are no memory
leaks.

To investigate more, I collected jemalloc stats after running the test for
few days and here is the summary for a run with narenas:1, tcache:false,
lg_chunk:24

 Arenas: 1
 Pointer size: 8
 Quantum size: 16
 Page size: 4096
 Min active:dirty page ratio per arena: 8:1
 Maximum thread-cached size class: 32768
 Chunk size: 16777216 (2^24)
 Allocated: 24364176040, active: 24578334720, mapped: 66739765248
 Current active ceiling: 24578621440
 chunks: nchunks   highchunks    curchunks
            3989         3978         3978
 huge: nmalloc      ndalloc    allocated
             3            2    117440512

 arenas[0]:
 assigned threads: 17
 dss allocation precedence: disabled
 dirty pages: 5971898:64886 active:dirty, 354265 sweeps, 18261119 madvises,
1180858954 purged

While in this state, the RSS of the process was at 54GB.

Questions:
1) The difference between RSS and jemalloc active is huge (more than 30GB).
In my test, the difference was quite less in the beginning (say 4 GB) and
it went on increasing with time. That seems too high to account for
jemalloc data structures, overhead etc. What else gets accounted in process
RSS - active?
2) The allocations are fairly random, sized between 8 bytes and 2MB. Are
there any known issues of fragmentation for particular allocation sizes?
3) Is there a way to tune the allocations and reduce the difference?

Will appreciate any help to understand this better,

Thanks,
Vandana
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jemalloc.net/mailman/jemalloc-discuss/attachments/20130422/774f20c4/attachment.html>