[concurrency-interest] Awaiting a set of tasks on an ExecutorService

Shevek shevek at anarres.org
Fri Aug 18 16:42:55 EDT 2017


I need to build a complicated object on a multiprocessor system. The 
target object uses synchronization to guarantee thread-safe access. 
However, the individual tasks which build elements of the target object 
are expensive, so I farm them out to an ExecutorService. The question is:

What is the recommended way to submit a stream of tasks to an 
ExecutorService, and then at a point, wait until they're all done.

* I don't know the list of tasks up front. I read them iteratively from 
a set of files or events.
* A big collection of Future(s) isn't realistic, there are just too many 
* Reaping completed futures on submission means that I end up with 
exceptions in weird places; I really need to gather max 100 suppressed 
exceptions, and discard the rest.
* ForkJoinPool has invoke() but I think for a million-task job, I still 
end up with a huge list of futures. This also assumes I ignore the note 
about synchronization.
* CompletionService allows me to wait for _any_ of the submitted tasks, 
but not for _all_ of them.

* Bonus points for sharing a single ExecutorService and having "sets" of 
tasks which can be independently awaited. This starts to sound very like 
FJP, except for the blocking/I/O and the stream-ness of my task set.

Mostly, dear list, I'm expecting a one-liner from some API class that 
I've missed because this is NOT an original request, I just can't find a 
standard solution.

Thank you.


More information about the Concurrency-interest mailing list