[concurrency-interest] ThreadLocal vs ProcessorLocal

Nathan Reynolds nathan.reynolds at oracle.com
Wed Oct 17 19:18:07 EDT 2012

Linux and Windows thread schedulers will migrate threads among cores on 
the same processor and to different processors very readily.  Solaris 
thread scheduler on Sparc really has to be pushed to move a thread to 
another processor.  The default for Xen virtual guests is to have the 
vCPUs assigned to a particular processor and the vCPUs can't migrate off 
that processor.  The vCPUs will bounce around the different cores on the 
same processor very readily, though.  Yes, load balancing can be 
problematic on Xen.  A test just bumped into that problem.

When building concurrent algorithms, data structures, locks, etc, one 
should definitely consider the core that the thread is currently running 
on.  However, the design should assume that the thread won't be running 
on that core or processor for very long.

When I built a NUMA-aware reader writer in C++, each reader acquires the 
lock tied to a particular core.  When the reader is done, it has to 
release that same lock even though it might be running on a different 
core.  This gave a huge scalability boost. We had 3 Nehalem processors 
100% utilized.  When we plugged in the 4^(t)^(h) Nehalem processor, it 
was 100% utilized but no increase in throughput.  When I make the reader 
writer lock NUMA-aware, then we got a significant boost in throughput 
from the 4^(t)^(h) Nehalem processor.

As for hot adding processors, this isn't too difficult.  Most data 
structures, locks, etc can handle it.  For example, simply add more 
leaves to the striping.  The changes don't have to be done efficiently 
or be concerned about scalability.  This is because hot adding 
processors should be a rare event in the lifetime of the process.

As for hot removing processors, this can be more difficult.  The only 
thing I could figure out is that the application has to notify the data 
structure when it is okay for it to deal with the removal.

For example, a NUMA-aware Exchanger wouldn't be bothered by hot plugged 
processors.  Sure, it will see an impact at the moment of the change, 
but it will then stabilize to the new processor configuration fairly 
quickly.  A NUMA-aware reader writer lock will require a bit more work 
but again it is doable.

Nathan Reynolds 
<http://psr.us.oracle.com/wiki/index.php/User:Nathan_Reynolds> | 
Consulting Member of Technical Staff | 602.333.9091
Oracle PSR Engineering <http://psr.us.oracle.com/> | Server Technology
On 10/17/2012 2:30 PM, Dr Heinz M. Kabutz wrote:
> [newbie question]
>     Is it possible for a thread to ever be moved between processors, 
> if for example the is an idle processor and another that is really 
> busy? If not, then load balancing could be problematic.  If yes, then 
> this means that we need to take this into consideration and not build 
> static structures based on the value of which processor the thread is 
> /currently/ running on.
> Again, if no, then what is the guarantee that it won't be available on 
> some VMs or some hardware architectures or operating systems?
> [/newbie question]
> Regards
> Heinz
> -- 
> Dr Heinz M. Kabutz (PhD CompSci)
> Author of "The Java(tm) Specialists' Newsletter"
> Sun Java Champion
> IEEE Certified Software Development Professional
> http://www.javaspecialists.eu
> Tel: +30 69 75 595 262
> Skype: kabutz
> On 10/17/12 11:05 AM, Romain Colle wrote:
>> On Wed, Oct 17, 2012 at 9:15 AM, Kirk Pepperdine <kirk at kodewerk.com 
>> <mailto:kirk at kodewerk.com>> wrote:
>>     SOT, right now we are using CPUID for a number of reasons. Very
>>     easy to get to on Linux as we can do this in pure Java. However
>>     other platforms require C++/assembler so having <gulp> an unsafe
>>     operation in the JVM would be a win from my POV.
>> I strongly second that. It is next to impossible to implement most 
>> NUMA-aware algorithms without knowing on which NUMA node the current 
>> thread is running.
>> Having this information available as a Java class (in Thread, Unsafe, 
>> ...) would be a huge win for people like us that need NUMA-aware 
>> behavior.
>> Cheers,
>> -- 
>> Romain Colle
>> R&D Project Manager
>> QuartetFS
>> 2 rue Jean Lantier, 75001 Paris, France
>> http://www.quartetfs.com <http://www.quartetfs.com/>
>> ------------------------------------------------------------------------
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
> _______________________________________________
> Concurrency-interest mailing list
> Concurrency-interest at cs.oswego.edu
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20121017/46feaa41/attachment.html>

More information about the Concurrency-interest mailing list