realloc stack use under O0 for x86_64

[Jason Evans]

On Mar 25, 2015, at 11:48 AM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
> We have a lightweight threading system where we are using very small stacks, on the order of 8-32k. We depend on jemalloc for scalable allocation.
> 
> When we compile jemalloc for debugging purposes, we see realloc using a large amount of stack space:
> 
> ```asm
> 0000000000070e48 <realloc>:
>   70e48:       55                      push   %rbp
>   70e49:       48 89 e5                mov    %rsp,%rbp
>   70e4c:       53                      push   %rbx
>   70e4d:       48 81 ec 68 62 00 00    sub    $0x6268,%rsp
> ```

Wow, that's far more stack space than I can imagine an explanation for.  What version of jemalloc is this happening with, which OS, compiler, etc.?

> [...]
> 
> I’ve tried to look through the source code for realloc, but I got a little lost. Can someone point me to what’s getting stack allocated that could possible need so much space? Is this a bug?

realloc() is actually je_realloc() in src/jemalloc.c.  In at least some release versions of jemalloc (not the current dev version), the fast path for realloc(NULL, size) is inlined, and that means a lot of functions could be involved.  Even so, none of them allocates huge on-stack data structures, and there's minimal recursive code, specifically in order to avoid the problems you're somehow hitting.

I can think of a couple possible techniques for narrowing down the problem.  One is to selectively force functions to not be inlined, and the other is to selectively disable compiler optimizations.  Neither is guaranteed to point directly at the problem though.

Thanks,
Jason