keeping memory usage at certain limit

[Antony Dovgal]

On 05/02/2014 08:37 AM, Jason Evans wrote:
> Your application can benefit from calling the “thread.tcache.flush” mallctl from a thread that is about to go “idle”
>(i.e. stops using the allocator for a while), but there’s little benefit otherwise, because there’s an incremental
>flushing mechanism built in that is driven by continued allocation activity.  One straightforward way to implement
>flushing for idle threads in thread pools is to have idle threads wake up after a few seconds of inactivity and flush before going back to sleep.

Ok, I got it, thank you for the explanation.

>> What buffles me most is that stats.allocated keeps returning the same value, but RSS constantly grows.
>
> This is probably because you aren’t calling the “epoch” mallctl to refresh mallctl’s cached statistics.

I just phrased it wrong: it does change (and I do update epoch).
When it goes above the limit I delete the old data and stats.allocated gets back to the value lower the limit again.
The problem is that even though stats.allocated is kept about the same value all the time, the difference between it and RSS continues to grow.
  
>> Could it be because of the amount of threads I use?
>
> If your application occasionally recurses deeply, you may be incrementally increasing the total amount of memory
> dedicated to thread execution stacks.  That could account for several gigabytes of memory usage, but probably isn’t the only issue.

I don't believe I have any recursion there.
Some libraries might use it though (Protobuf and Judy come to my mind first), I'll check it out, thanks for the suggestion.

>> Say, I free memory in one thread and try to allocate in another one, but the second thread
>> doesn't have it cached and has to do the actual allocation?
>
> Within limits, this can bloat memory usage.  However, IIRC thread caches average ~2.5 MiB per thread under the worst conditions
>(all threads are purely deallocating a broad mix of allocation sizes), so the thread caches probably account for less than 1 GiB in your application.

*sigh* now I'm out of theories.. =)

-- 
Wbr,
Antony Dovgal
---
http://pinba.org - realtime profiling for PHP