[concurrency-interest] High CPU utilization on CentOS6 in TPE using LinkedTransferQueue

Serguei Mourachov smourachov at gmail.com
Mon Jul 16 20:55:04 EDT 2012


We are having problem migrating our system from CentOS 5.5(Linux kernel 
version 2.6.18) to CentOS 6 (kernel version 2.6.32) running JDK7.
We are using Amazon EC2 c1.xlarge instance with 8 virtual cores.
Here is the code snippet to reproduce the problem, simulating our workload :

public static void main(String[] args) throws Exception {
         ThreadMXBean threadMxBean = ManagementFactory.getThreadMXBean();
         Runnable r = new Runnable() {
             public void run() {
                 LockSupport.parkNanos(1_000_000);
             }
         };
         ThreadPoolExecutor pool = new ThreadPoolExecutor(128, 128, 0, 
TimeUnit.MINUTES, new LinkedTransferQueue<Runnable>());
         long cpuTotalTimeOnStart = threadMxBean.getCurrentThreadCpuTime();

         for (int i = 0; i < 2_000_000; i++) {
             pool.execute(r);
         }
         long cpuTotalTime = (threadMxBean.getCurrentThreadCpuTime() - 
cpuTotalTimeOnStart) / 1_000_000;
         System.out.println(" cpuTotalTime=" + cpuTotalTime + "ms");
         System.exit(0);
     }
}
When executed on CentOS 5.5 the main thread is consuming <300ms of cpu 
time, but in  case of CentOS 6 that value is growing to >15s of cpu time,
We traced that high cpu utilization down to LockSupport.unpark() calls 
in LinkedTransferQueue.xfer() method.
If we replace LinkedTransferQueue with "traditional" 
LinkedBlockingQueue, then the cpu utilization becomes comparable to 
CentOS 5.5, but we really like much better latency of LTQ.
Decreasing TPE size to 64 threads also brings cpu utilization down to 
<500ms values, but this is not a feasible solution in case of our system.
Apparently,  the issue is caused by switching to "Completely Fair 
Scheduler"  and, in case of  CentOS 6, 80-90% of total cpu time is the 
system time.
A the moment we are trying to play with some kernel scheduling 
parameters,  but without any major success, so any help or advice will 
be highly appreciated

Serguei Mourachov


More information about the Concurrency-interest mailing list