arenas.extend + thread.arena confusion

[Jason Evans]

On Sep 30, 2014, at 11:08 AM, D'Alessandro, Luke K <ldalessa at indiana.edu> wrote:
> I have an application where I want every thread to have two arenas. One is use for default allocations and has access to the cache, and the other is used for private allocations through malllocx().
> 
> I do this by doing an arena.extend + thread.arena in each thread. The problem that I have is that jemalloc seems to reuse arena ids in this context. Essentially I get a trace that looks something like:
> 
> t1: t2 = pthread_create()
> t1: new1 = arenas.extend
> t1: old1 = thread.arena(new1)
> t2: new2 = arenas.extend
> t2: old2 = thread.arena(new2)
> 
> : old1 == old2 == 0
> 
> Is this behavior expected? Shouldn’t jemalloc use a fresh arena for each new thread?

When jemalloc assigns an arena to a thread, it finds the set of default (non-"extend") arenas that have the fewest assigned threads, and assigns the lowest-numbered arena in that set to the thread.  If you were to remove the "thread.arena" assignment from your test program (which is consistent with your stated purpose), you would end up with your threads assigned to arenas 0 and 1, and you would additionally have two arenas that jemalloc uses only if you specify MALLOCX_ARENA() to one of the *allocx() functions.  As it is, the test program is racey; you could end up with the created thread initially assigned to arena 1, if a context switch happened at the right time.

Jason