Profiling memory allocations in run-time in production

[Jason Evans]

On Dec 22, 2013, at 11:41 PM, Evgeniy Ivanov <i at eivanov.com> wrote:
> I need to profile my application running in production. Is it
> performance safe to build jemalloc with "--enable-prof", start
> application with profiling disabled and enable it for short time
> (probably via mallctl() call), when I need? I'm mostly interested in
> stacks, i.e. opt.prof_accum. Or are there better alternatives in
> Linux? I've tried perf, but it just counts stacks and doesn't care
> about amount of memory allocated. There is also stap, but I haven't
> try it yet.

Yes, you can use jemalloc's heap profiling as you describe, with essentially no performance impact while heap profiling is inactive.  You may even be able to leave heap profiling active all the time with little performance impact, depending on how heavily your application uses malloc.  At Facebook we leave heap profiling active all the time for a wide variety of server applications; there are only a couple of exceptions I'm aware of for which the performance impact is unacceptable (heavy malloc use, ~2% slowdown when heap profiling is active).

Jason