Hi,
I am observing a huge gap between the "total allocated" &
"active" counts in the jemalloc stats. The "active" & "mapped"
correctly point to the RSS and VIRT counters in top. Below
is a snippet of the stats output.
How should I infer this gap? Is this the fragmentation caused
by the chunk metadata & unused dirty pages? I am purging unused
dirty pages a bit more aggressively than default (lg_dirty_mult: 5).
Should I consider being more aggressive?
Secondly, I am using 1 arena per CPU core but my application creates
lots of transient threads making small allocations. Should I consider
using more arenas to mitigate performance bottlenecks incurred due to
blocking on per-arena locks?
Finally, looking at the jemalloc stats how should I go about
configuring the tcache? My application has a high thread churn &
each thread performs lots of short-lived small allocations. Should
I consider decreasing lg_tcache_max to 4K?
Thanks in advance for your help & a great !
Nikhil
Version: 3.4.1-0-g0135fb806e4137dc9cdf152541926a2bc95e33f0
Assertions disabled
Run-time option settings:
opt.abort: false
opt.lg_chunk: 22
opt.dss: "secondary"
opt.narenas: 32
opt.lg_dirty_mult: 5
opt.stats_print: false
opt.junk: false
opt.quarantine: 0
opt.redzone: false
opt.zero: false
opt.tcache: true
opt.lg_tcache_max: 15
CPUs: 32
Arenas: 32
Pointer size: 8
Quantum size: 16
Page size: 4096
Min active:dirty page ratio per arena: 32:1
Maximum thread-cached size class: 32768
Chunk size: 4194304 (2^22)
Allocated: 4384464272, active: 7056244736, mapped: 9684647936
Current active ceiling: 7117733888
chunks: nchunks highchunks curchunks
7025 2310 2309
huge: nmalloc ndalloc allocated
2049 2049 0
Merged arenas stats:
assigned threads: 2000
dss allocation precedence: N/A
dirty pages: 1722716:44009 active:dirty, 7058804 sweeps, 58038956 madvises, 213929279 purged
allocated nmalloc ndalloc nrequests
small: 3511692688 47323002828 47265541258 113947055682
large: 872771584 173173680 173125745 396022674
total: 4384464272 47496176508 47438667003 114343078356
active: 7056244736
mapped: 9680453632
bins: bin size regs pgs allocated nmalloc ndalloc nrequests nfills nflushes newruns reruns curruns
0 8 501 1 50937728 9040731301 9034364085 17223271472 616102986 95651203 1077826 73949677 40745
1 16 252 1 77020144 573171274 568357515 2242686659 96351930 10266723 257739 44306633 21604
2 32 126 1 429852096 18555361191 18541928313 39525242385 509036325 189701168 30245928 575288258 231731
3 48 84 1 774254160 13193445786 13177315491 32755947543 645431203 161130307 11413376 719379426 344983
4 64 63 1 270561344 1606583978 1602356457 7022116344 175867804 29834157 2200328 286059147 102283
5 80 50 1 526179760 844027433 837450186 3673315871 80016507 20912400 3115050 123785057 163248
6 96 84 2 66918048 643271050 642573987 2261869920 94406420 13502497 378473 46646128 20469
7 112 72 2 141823360 613974631 612708351 1586692517 81034870 14137080 1211302 49616234 31895
8 128 63 2 117911808 261090749 260169563 1556337221 34923763 8789674 1200493 28901458 22666
9 160 51 2 104119200 623493227 622842482 2021853850 74126800 17569428 3034778 62741358 22748
10 192 63 3 178081344 483607033 482679526 904009644 182288927 13367164 684876 23522728 20630
11 224 72 4 65155104 106499296 106208425 332165705 31167068 5082954 439625 6081887 5327
12 256 63 4 48990208 175064106 174872738 472273979 16503128 7906539 199089 10246092 7009
13 320 63 5 99602240 183354471 183043214 372355472 51938313 7061197 441119 20284908 10444
14 384 63 6 22376448 44600424 44542152 120348652 10768940 3882450 397080 3666681 1897
15 448 63 7 19032384 182663513 182621030 217533670 23470929 6906695 1008007 4392356 2290
16 512 63 8 83511808 60243213 60080104 1082410264 26644419 7034468 173287 10135623 4852
17 640 51 8 40183040 36188548 36125762 161999704 6264681 4081641 43579 4966597 2979
18 768 47 9 17687040 3052902 3029872 10607471 1848614 1850140 2010 1170181 747
19 896 45 10 17929856 4151025 4131014 22233426 2621174 1978063 1738 1655723 730
20 1024 63 16 226070528 51727670 51506898 181066689 13752482 5460833 33701 6304216 4142
21 1280 51 16 24062720 6708042 6689243 34800612 3560278 3026925 2015 2823329 786
22 1536 42 16 9480192 5591142 5584970 96743012 4922706 2550424 12388 897135 326
23 1792 38 17 3695104 17709685 17707623 49250101 6896535 2060250 322840 255016 223
24 2048 65 33 42412032 3559112 3538403 12755832 2165621 1725408 13807 138715 565
25 2560 52 33 27392000 1716475 1705775 4200713 1311249 1165105 1082 37596 760
26 3072 43 33 1959936 491695 491057 810373 359939 339356 97162 3042 65
27 3584 39 35 24493056 923856 917022 2156581 684008 693906 2718 20676 235
large: size pages nmalloc ndalloc nrequests curruns
4096 1 3573958 3566123 56996547 7835
8192 2 126515167 126505796 204913885 9371
12288 3 5594725 5589477 17350232 5248
16384 4 3762445 3757246 20675309 5199
20480 5 3241286 3229987 4354437 11299
24576 6 22339006 22336848 65613502 2158
28672 7 343822 342626 346351 1196
32768 8 1572379 1569012 19541519 3367
36864 9 2708929 2708708 2708929 221
40960 10 32955 32952 32955 3
45056 11 57426 57419 57426 7
49152 12 583663 583651 583663 12
53248 13 54411 54368 54411 43
57344 14 141737 141665 141737 72
61440 15 1660 1522 1660 138
65536 16 1422733 1422535 1422733 198
69632 17 13344 13196 13344 148
73728 18 2949 2810 2949 139
77824 19 1533 1356 1533 177
81920 20 342699 342484 342699 215
86016 21 1561 1353 1561 208
90112 22 2487 2389 2487 98
94208 23 1456 1329 1456 127
98304 24 17666 17593 17666 73
102400 25 4889 4798 4889 91
106496 26 1410 1329 1410 81
110592 27 1414 1296 1414 118
114688 28 500876 500818 500876 58
118784 29 1223 1216 1223 7
122880 30 1528 1528 1528 0
126976 31 1202 1202 1202 0
131072 32 55098 55096 55098 2
135168 33 1292 1292 1292 0
139264 34 1218 1218 1218 0
143360 35 1221 1221 1221 0
147456 36 1192 1192 1192 0
151552 37 1214 1214 1214 0
155648 38 1244 1244 1244 0
159744 39 6360 6360 6360 0
163840 40 2367 2367 2367 0
167936 41 3538 3538 3538 0
172032 42 1206 1206 1206 0
176128 43 2271 2271 2271 0
180224 44 4303 4303 4303 0
184320 45 1220 1220 1220 0
188416 46 1177 1177 1177 0
192512 47 1182 1182 1182 0
196608 48 1305 1305 1305 0
200704 49 4508 4496 4508 12
204800 50 25259 25258 25259 1
208896 51 3380 3380 3380 0
212992 52 1335 1335 1335 0
217088 53 307 307 307 0
221184 54 309 309 309 0
225280 55 307 307 307 0
229376 56 3383 3383 3383 0
233472 57 308 308 308 0
237568 58 306 306 306 0
241664 59 308 308 308 0
245760 60 305 305 305 0
249856 61 302 302 302 0
253952 62 304 304 304 0
258048 63 314 314 314 0
262144 64 21397 21397 21397 0
266240 65 306 305 306 1
270336 66 304 304 304 0
274432 67 304 304 304 0
278528 68 292 292 292 0
282624 69 274 274 274 0
286720 70 273 273 273 0
290816 71 271 271 271 0
294912 72 29417 29417 29417 0
299008 73 275 275 275 0
303104 74 273 273 273 0
307200 75 273 273 273 0
311296 76 270 270 270 0
315392 77 273 273 273 0
319488 78 2319 2319 2319 0
323584 79 274 274 274 0
327680 80 1304 1304 1304 0
331776 81 2626 2626 2626 0
335872 82 274 274 274 0
339968 83 273 273 273 0
344064 84 275 275 275 0
348160 85 270 270 270 0
352256 86 273 273 273 0
356352 87 274 274 274 0
360448 88 3342 3342 3342 0
364544 89 273 273 273 0
368640 90 270 270 270 0
372736 91 274 274 274 0
376832 92 270 270 270 0
380928 93 272 272 272 0
385024 94 272 272 272 0
389120 95 272 272 272 0
393216 96 276 276 276 0
397312 97 290 281 290 9
401408 98 273 273 273 0
405504 99 270 270 270 0
409600 100 9279 9278 9279 1
413696 101 5142 5142 5142 0
417792 102 3089 3089 3089 0
421888 103 5181 5181 5181 0
425984 104 47539 47539 47539 0
430080 105 17 17 17 0
434176 106 15 15 15 0
438272 107 17 17 17 0
442368 108 14 14 14 0
446464 109 17 17 17 0
450560 110 20 20 20 0
454656 111 15 15 15 0
458752 112 16 16 16 0
462848 113 15 15 15 0
466944 114 16 16 16 0
471040 115 17 17 17 0
475136 116 14 14 14 0
479232 117 17 17 17 0
483328 118 15 15 15 0
487424 119 16 16 16 0
491520 120 16 16 16 0
495616 121 17 17 17 0
499712 122 14 14 14 0
503808 123 19 19 19 0
507904 124 17 17 17 0
512000 125 14 14 14 0
516096 126 17 17 17 0
520192 127 15 15 15 0
524288 128 35043 35043 35043 0
528384 129 14 14 14 0
532480 130 16 16 16 0
536576 131 16 16 16 0
540672 132 15 15 15 0
544768 133 16 16 16 0
548864 134 14 14 14 0
552960 135 16 16 16 0
557056 136 2 2 2 0
[2]
569344 139 1 1 1 0
573440 140 1 1 1 0
[1]
581632 142 1 1 1 0
[1]
589824 144 1 1 1 0
[1]
598016 146 1 1 1 0
[3]
614400 150 1 1 1 0
[1]
622592 152 1 1 1 0
[5]
647168 158 1 1 1 0
651264 159 2 2 2 0
655360 160 3 3 3 0
[3]
671744 164 2 2 2 0
675840 165 1 1 1 0
[2]
688128 168 1 1 1 0
708608 173 1 1 1 0
[2]
720896 176 5122 5122 5122 0
[11]
770048 188 1 1 1 0
[2]
782336 191 1 1 1 0
786432 192 1 1 1 0
790528 193 2 2 2 0
[1]
798720 195 1 1 1 0
[4]
819200 200 1 1 1 0
[1]
827392 202 2048 2048 2048 0
831488 203 5120 5120 5120 0
[1]
839680 205 7196 7196 7196 0
[15]
905216 221 1 1 1 0
[1]
913408 223 1 1 1 0
[13]
970752 237 1 1 1 0
[12]
1024000 250 1 1 1 0
[1]
1032192 252 1 1 1 0
[4]
1052672 257 1 1 1 0
[48]
1253376 306 1 1 1 0
1257472 307 1 1 1 0
[4]
1277952 312 1 1 1 0
[7]
1310720 320 2 2 2 0
[7]
1343488 328 1 1 1 0
1347584 329 1 1 1 0
[15]
1413120 345 1 1 1 0
1417216 346 1 1 1 0
[4]
1437696 351 2048 2048 2048 0
[9]
1478656 361 1 1 1 0
[14]
1540096 376 1 1 1 0
[2]
1552384 379 1 1 1 0
1576960 385 2 1 2 1
[20]
1662976 406 2048 2048 2048 0
[3]
1679360 410 2048 2048 2048 0
[21]
1769472 432 1626 1626 1626 0
[2]
1781760 435 1 1 1 0
[146]
2383872 582 7 7 7 0
2387968 583 1016 1016 1016 0
[105]
2822144 689 1 1 1 0
[79]
3149824 769 1 0 1 1
[168]
3842048 938 1 1 1 0
[80]