[concurrency-interest] ForkJoinPool creating thousands of threads

Christopher Berner christopher at trumpet.io
Tue Feb 22 17:21:37 EST 2011


Hi everyone,

I recently started using ForkJoin for a large numerical analysis job. I've subclassed RecursiveAction, and implemented compute() essentially as follows

if (users.size <= THRESHOLD) {
  System.out.println("Computing");
  //do work
  System.out.println("Computation took X time.");
} else {
  MyAction right = new MyAction(users.subList(0, THRESHOLD));
  MyAction left = new MyAction(users.subList(THRESHOLD, users.size()));
  left.fork();
  right.compute();
  System.out.println("Joining");
  left.join();
  //do a little more work
}


The results I'm seeing are very confusing, however. It appears that the threads are blocking on join, instead of stealing more work. And the problem appears to get worse over time, as after ~30secs the pool has spawned thousands of threads, and almost all of them are "waiting." If anyone knows why this is happening, I'd very much appreciate any insight you can share.

Thanks,
Christopher


Below is part of the logged output:

INFO 2011-02-22 14:10:39,264 ForkJoinPool-1-worker-0  Starting compute
INFO 2011-02-22 14:10:39,264 ForkJoinPool-1-worker-3  Starting compute
INFO 2011-02-22 14:10:39,266 ForkJoinPool-1-worker-2  Starting compute
INFO 2011-02-22 14:10:39,265 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,275 ForkJoinPool-1-worker-3  96 items in 9.248ms
INFO 2011-02-22 14:10:39,275 ForkJoinPool-1-worker-3  Joining
<<< Note that worker 3 never shows up again >>>
INFO 2011-02-22 14:10:39,283 ForkJoinPool-1-worker-2  58 items in 16.487ms
INFO 2011-02-22 14:10:39,283 ForkJoinPool-1-worker-2  Joining
INFO 2011-02-22 14:10:39,283 ForkJoinPool-1-worker-2  Starting compute
INFO 2011-02-22 14:10:39,286 ForkJoinPool-1-worker-2  34 items in 2.744ms
INFO 2011-02-22 14:10:39,286 ForkJoinPool-1-worker-2  Joining
INFO 2011-02-22 14:10:39,286 ForkJoinPool-1-worker-2  Starting compute
INFO 2011-02-22 14:10:39,287 ForkJoinPool-1-worker-1  55 items in 20.468ms
INFO 2011-02-22 14:10:39,287 ForkJoinPool-1-worker-1  Joining
INFO 2011-02-22 14:10:39,287 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,290 ForkJoinPool-1-worker-1  47 items in 3.087ms
INFO 2011-02-22 14:10:39,290 ForkJoinPool-1-worker-1  Joining
INFO 2011-02-22 14:10:39,290 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,302 ForkJoinPool-1-worker-0  39 items in 36.166ms
INFO 2011-02-22 14:10:39,302 ForkJoinPool-1-worker-0  Joining
INFO 2011-02-22 14:10:39,303 ForkJoinPool-1-worker-1  54 items in 13.156ms
INFO 2011-02-22 14:10:39,303 ForkJoinPool-1-worker-1  Joining
.
.
.
<<< The pool spawns a fifth worker (my computer only has 4 cores) >>>
INFO 2011-02-22 14:10:39,527 ForkJoinPool-1-worker-4  Starting compute
INFO 2011-02-22 14:10:39,527 ForkJoinPool-1-worker-2  39 items in 1.129ms
INFO 2011-02-22 14:10:39,527 ForkJoinPool-1-worker-2  Joining
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-4  49 items in 1.109ms
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-4  Joining
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-4  Starting compute
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-1  43 items in 1.857ms
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-1  Joining
INFO 2011-02-22 14:10:39,528 ForkJoinPool-1-worker-1  Starting compute
INFO 2011-02-22 14:10:39,529 ForkJoinPool-1-worker-4  41 items in 1.019ms
.
.
.
<<< A second later we're already up to 22 workers >>>
INFO 2011-02-22 14:10:40,578 ForkJoinPool-1-worker-22  16 items in 11.961ms
INFO 2011-02-22 14:10:40,578 ForkJoinPool-1-worker-22  Joining
INFO 2011-02-22 14:10:40,578 ForkJoinPool-1-worker-22  Starting compute
INFO 2011-02-22 14:10:40,586 ForkJoinPool-1-worker-21  18 items in 11.595ms
INFO 2011-02-22 14:10:40,586 ForkJoinPool-1-worker-21  Joining
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20110222/72efe4f4/attachment.html>


More information about the Concurrency-interest mailing list