[concurrency-interest] ForkJoinPool creating thousands of threads

Geert Denys gdenys at yahoo.com
Wed Feb 23 04:30:44 EST 2011


As each task only forks one other task, no additional parallellism can be 
achieved by the FJ pool and it's basically a sequential execution. Splitting the 
list in two equal parts and forking tasks for those parts should be a better 
match for FJ.

Regards,
Geert.



________________________________
From: Christopher Berner <christopher at trumpet.io>
To: concurrency-interest at cs.oswego.edu
Sent: Tue, February 22, 2011 11:21:37 PM
Subject: [concurrency-interest] ForkJoinPool creating thousands of threads

Hi everyone,

I recently started using ForkJoin for a large numerical analysis job. I've 
subclassed RecursiveAction, and implemented compute() essentially as follows

if (users.size <= THRESHOLD) {
  System.out.println("Computing");
  //do work
  System.out.println("Computation took X time.");
} else {
  MyAction right = new MyAction(users.subList(0, THRESHOLD));
  MyAction left = new MyAction(users.subList(THRESHOLD, users.size()));
  left.fork();
  right.compute();
  System.out.println("Joining");
  left.join();
  //do a little more work
}


The results I'm seeing are very confusing, however. It appears that the threads 
are blocking on join, instead of stealing more work. And the problem appears to 
get worse over time, as after ~30secs the pool has spawned thousands of threads, 
and almost all of them are "waiting." If anyone knows why this is happening, I'd 
very much appreciate any insight you can share.

Thanks,
Christopher


Below is part of the logged output:

INFO 2011-02-22 14:10:39,264 ForkJoinPool-1-worker-0  Starting compute
INFO 2011-02-22 14:10:39,264 ForkJoinPool-1-worker-3  Starting compute
INFO 2011-02-22 14:10:39,266 ForkJoinPool-1-worker-2  Starting compute
INFO 2011-02-22 14:10:39,265 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,275 ForkJoinPool-1-worker-3  96 items in 9.248ms
INFO 2011-02-22 14:10:39,275 ForkJoinPool-1-worker-3  Joining
<<< Note that worker 3 never shows up again >>>
INFO 2011-02-22 14:10:39,283 ForkJoinPool-1-worker-2  58 items in 16.487ms
INFO 2011-02-22 14:10:39,283 ForkJoinPool-1-worker-2  Joining
INFO 2011-02-22 14:10:39,283 ForkJoinPool-1-worker-2  Starting compute
INFO 2011-02-22 14:10:39,286 ForkJoinPool-1-worker-2  34 items in 2.744ms
INFO 2011-02-22 14:10:39,286 ForkJoinPool-1-worker-2  Joining
INFO 2011-02-22 14:10:39,286 ForkJoinPool-1-worker-2  Starting compute
INFO 2011-02-22 14:10:39,287 ForkJoinPool-1-worker-1  55 items in 20.468ms
INFO 2011-02-22 14:10:39,287 ForkJoinPool-1-worker-1  Joining
INFO 2011-02-22 14:10:39,287 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,290 ForkJoinPool-1-worker-1  47 items in 3.087ms
INFO 2011-02-22 14:10:39,290 ForkJoinPool-1-worker-1  Joining
INFO 2011-02-22 14:10:39,290 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,302 ForkJoinPool-1-worker-0  39 items in 36.166ms
INFO 2011-02-22 14:10:39,302 ForkJoinPool-1-worker-0  Joining
INFO 2011-02-22 14:10:39,303 ForkJoinPool-1-worker-1  54 items in 13.156ms
INFO 2011-02-22 14:10:39,303 ForkJoinPool-1-worker-1  Joining
.
.
.
<<< The pool spawns a fifth worker (my computer only has 4 cores) >>>
INFO 2011-02-22 14:10:39,527 ForkJoinPool-1-worker-4  Starting compute
INFO 2011-02-22 14:10:39,527 ForkJoinPool-1-worker-2  39 items in 1.129ms
INFO 2011-02-22 14:10:39,527 ForkJoinPool-1-worker-2  Joining
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-4  49 items in 1.109ms
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-4  Joining
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-4  Starting compute
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-1  43 items in 1.857ms
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-1  Joining
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,529 ForkJoinPool-1-worker-4  41 items in 1.019ms
.
.
.
<<< A second later we're already up to 22 workers >>>
INFO 2011-02-22 14:10:40,578 ForkJoinPool-1-worker-22  16 items in 11.961ms
INFO 2011-02-22 14:10:40,578 ForkJoinPool-1-worker-22  Joining
INFO 2011-02-22 14:10:40,578 ForkJoinPool-1-worker-22  Starting compute
INFO 2011-02-22 14:10:40,586 ForkJoinPool-1-worker-21  18 items in 11.595ms
INFO 2011-02-22 14:10:40,586 ForkJoinPool-1-worker-21  Joining


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20110223/41327aed/attachment.html>


More information about the Concurrency-interest mailing list