[concurrency-interest] Thread exhaustion
dl at cs.oswego.edu
Tue Apr 6 06:49:27 EDT 2010
On 04/06/10 03:51, Alexey Kudravtsev wrote:
> I am now confronting the simple task of mass parallelism that should
> have been a perfect fit for FJ.
> So I wrote the program to simulate this:
Could you send me (off-list) the full version of this so
I can diagnose?
> Basically, each ParallelArray.apply() task calls join()
> which blocks the current thread until there are no threads left)
> If I comment out pool.setMaximumPoolSize() line, it kinda works, but
> generate excessive number of 'spare' threads.
(It is rarely a good idea to setMaximumSize in FJ.)
> The snippet above allocated 64 threads on my machine, and my real data
> sets caused allocation of hundreds(!) spare threads, most of which do
> nothing just waiting.
> So I figured that replacing join() in ParallelArray.apply() with
> helpJoin() could help.
> Unfortunately, the program still deadlocks even then. As far as I
> understand, the problem is that helpJoin() tries to steal tasks from
> other worker thread queue, and,
> failing that, blocks hard, waiting for completion of its own task. There
> is a possibility that all worker thread queues went empty for a moment,
> and in this instant the helpJoin() will block its current thread.
Could you please try both join- and helpJoin- based versions
using the jsr166y update (from Apr 5+)? It includes improved spare
thread throttling for join and also an improved form of helpJoin
that falls back to thread creation if needed.
> P.S. I think the fact that ParallelArray tasks use join() instead of
> helpJoin() or equivalent makes them not very useful.
> All of the ParallelArray tasks are independent and therefore are
> perfectly eligible for helpJoining each other.
ParallelArray cannot be sure that a computation does
not itself start another independent FJ computation,
that would require join, not helpJoin to work well.
It may be possible to adapt ParallelArray methods to be
a little smarter about this though.
More information about the Concurrency-interest