Does arena_tcache_fill_small() ever end up bypassing custom chunk allocation?

[Jason Evans]

On Feb 10, 2015, at 8:29 PM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
>> On Feb 10, 2015, at 11:00 PM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
>>> On Feb 10, 2015, at 10:52 PM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
>>> We have an arena that we are using for a specific bit of memory that we are managing with a custom chunk allocator.
>>> 
>>> We’re seeing initial small allocations through this arena succeed without calling out to our custom chunk allocator. This behavior appears to be new since b617df8. While tracing this behavior, I’m seeing the initial allocations miss in the cache, and forward to arena_tcache_fill_small() which does some difficult-to-disect stuff including something to do with runs. After this call, the cache succeeds in supplying the allocation, without ever getting our chunk allocator involved.

10aff3f3e1b8b3ac0348b259c439c9fe870a6b95 had a lot of a0-related changes in it, which may be related to the behavior change you're seeing.

>>> This causes us issues because the memory we’re getting back doesn’t seem to come from the place we’re expecting it to come from.
>>> 
>>> Is arena_tcache_fill_small() going somewhere special to find memory that was allocated with some other chunk allocator, previous to our initialization of our arena?
>>> 
>> Ah, actually, I see one call happening to the default chunk allocator for another arena during initialization.
>> 
>> We’re calling mallctl with opt.lg_chunk, and that triggers a really early a0malloc() inside of ctl_init(), which cascades into the chunk_alloc_default before we’ve had a chance to set our custom allocator on the default arena. I guess that arena_tcache_fill_small() uses this chunk to satisfy misses for small objects, which makes sense.

Is the problem that the region that is allocated from arena 0 ends up being freed back to the tcache, and you're depending on the tcache only containing regions from your custom arena?  In that case you could flush the tcache once you have the arena fully configured.

>> Is this early a0malloc() behavior new? 
>> 
>> I’ll will try and see if we can promote our custom allocator to be the default arena’s allocator before we do any other interaction with jemalloc.
>> Let me know if there is any other way to bypass this issue,
> 
> Okay, I’ve run into a wall here.
> 
> In order to change the custom chunk allocator I am using the mallctl interface, which requires that I provide the arena id which I normally get through mallctl, but this triggers the a0malloc() that causes my problem.
> 
> If I hard-code arena to be 0, which is correct, but feels brittle, I can skip that initial mallctl query, however I still need to use mallctl to set the chunk allocator which will trigger the bad a0malloc() before the allocator is installed.
> 
> Is there a way around this? Could I set the allocator without passing through the mallctl interface as a workaround for now? Where would I look to do that? Can I have a0malloc() use the system malloc instead (we’re using jemalloc in prefixed form)?
> 
> Any help would be appreciated as this is a blocker for us.

I'm a bit confused about what your code is trying to do, but maybe you're hitting one of these problems:

- You're trying to change the chunk allocator for one of the automatic arenas (arena 0?).  This is unlikely to ever work reliably for arena 0, though it would potentially be possible for other automatic arenas prior to launching any threads.
- You're calling the thread.arena mallctl to refer to a newly created arena before you've finished setting up the arena to use your chunk allocator.

I still have a bit of work to do on making sure that no metadata are allocated from non-auto arenas, so that they can be reset (see https://github.com/jemalloc/jemalloc/issues/146).  Is that related to what you're hitting?

Thanks,
Jason