[concurrency-interest] Awaiting a set of tasks on an ExecutorService

Josh Humphries jhump at bluegosling.com
Fri Aug 18 16:53:27 EDT 2017


I forgot to address the concern about error handling. For that, you could
collect them into a synchronized list, where you simply ignore failures
after the size of the list reaches the limit.

Another possibility is to use an ExecutorCompletionService. You can create
a new one for each set of tasks, submit all tasks to it, and then poll them
all to wait until tasks finish. With that approach, you won't need to
decorate the tasks, and instead handle the errors and tracking how many
tasks you've polled in the code that is waiting for them to finish.

The ExecutorCompletionService also lets you re-use an ExecutorService. You
create a new CompletionService for each group of tasks, but they can all
wrap the same underlying ExecutorService.


----
*Josh Humphries*
jhump at bluegosling.com

On Fri, Aug 18, 2017 at 4:48 PM, Josh Humphries <jhump at bluegosling.com>
wrote:

> I think the easiest thing would be to decorate each task to call
> "latch.countDown()" on a CountDownLatch that is initialized with the total
> number of tasks. After they are all submitted, the code that wants to wait
> for them to finish would simply await the latch. This decouples completion
> of all tasks from the actual ExecutorService that is running them, so you
> can share the same ExecutorService for multiple, even overlapping, sets of
> tasks.
>
> ----
> *Josh Humphries*
> jhump at bluegosling.com
>
> On Fri, Aug 18, 2017 at 4:42 PM, Shevek <shevek at anarres.org> wrote:
>
>> Hi,
>>
>> I need to build a complicated object on a multiprocessor system. The
>> target object uses synchronization to guarantee thread-safe access.
>> However, the individual tasks which build elements of the target object are
>> expensive, so I farm them out to an ExecutorService. The question is:
>>
>> What is the recommended way to submit a stream of tasks to an
>> ExecutorService, and then at a point, wait until they're all done.
>>
>> * I don't know the list of tasks up front. I read them iteratively from a
>> set of files or events.
>> * A big collection of Future(s) isn't realistic, there are just too many
>> tasks.
>> * Reaping completed futures on submission means that I end up with
>> exceptions in weird places; I really need to gather max 100 suppressed
>> exceptions, and discard the rest.
>> * ForkJoinPool has invoke() but I think for a million-task job, I still
>> end up with a huge list of futures. This also assumes I ignore the note
>> about synchronization.
>> * CompletionService allows me to wait for _any_ of the submitted tasks,
>> but not for _all_ of them.
>>
>> * Bonus points for sharing a single ExecutorService and having "sets" of
>> tasks which can be independently awaited. This starts to sound very like
>> FJP, except for the blocking/I/O and the stream-ness of my task set.
>>
>> Mostly, dear list, I'm expecting a one-liner from some API class that
>> I've missed because this is NOT an original request, I just can't find a
>> standard solution.
>>
>> Thank you.
>>
>> S.
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20170818/f7ae28c6/attachment.html>


More information about the Concurrency-interest mailing list