[concurrency-interest] Exception handling & Executors

Joe Bowbeer joe.bowbeer at gmail.com
Tue Oct 25 16:21:40 EDT 2005


I meant:

executor.execute(runnable)

This gives you the most control over the actual task that is
executing.  For example, your task could be a custom FutureTask with a
done() method specialized for handling OOME.

To isolate the impact, you could create a "task factory" that took a
runnable and returned a FutureTask customized for OOME handling.

Or (and this is probably the best solution for you) you could create a
custom ExecutorService that did this for you, by extending
AbstractExecutorService.

This sort of extension is made even easier in 1.6 Mustang with the
addition of the newTaskFor method:

    protected <T> RunnableFuture<T> newTaskFor(Runnable runnable, T value) {
        return new FutureTask<T>(runnable, value);
    }


On 10/25/05, John Adams <jadams3 at nortel.com> wrote:
> Hi Joe,
>
> Thank you for the reply.  Regarding your comment about OutOfMemory, as
> was buried in the original post, I want to log the exception and then do
> a System.exit().  We have a watchdog process that restarts the server
> and starts again.  If we have a gradual leak going on, it's better in
> the field for us to raise an alarm and restart the server than it is to
> thrash with out of memory.  We're trying to build a reliable system, and
> we have to deal with the possibility that this could happen in the
> field, despite all of our best leak detection efforts.
>
> To respond ...
>
> 1. We use runnables in our implementation already.  If you pass a
> runnable into a task, this code in AbstractExecutorService gets hit ...
>
>  public <T> Future<T> submit(Runnable task, T result) {
>         if (task == null) throw new NullPointerException();
>         FutureTask<T> ftask = new FutureTask<T>(task, result);
>         execute(ftask);
>         return ftask;
>     }
>
> Note the wrapping of the FutureTask, and the source of my complaint.
> The inner task uses Sync, an inner class of FutureTask, which does this
> ..
>
>  void innerRun() {
>             if (!compareAndSetState(0, RUNNING))
>                 return;
>             try {
>                 runner = Thread.currentThread();
>                 innerSet(callable.call());
>             } catch(Throwable ex) {
>                 innerSetException(ex);
>             }
>         }
>
> 2. I do hook into the afterExecute() which resolves my problem, albeit
> with some ugly code.   I'm really just saying that this solution is
> complex and hoping someone in the future will implement a
> setDefaultExceptionHandler() equivalent for the Executor framework.  I
> also thought others might have this problem, so it would be useful to
> provide this code to others with similar requirements.
>
> 3. The Thread.setDefaultExceptionHandler doesn't resolve the problem,
> the FutureTask/Sync catches throwable.
>
> 4. The done() idea is interesting, although I'm guessing that's part of
> how the custom future task will deal with the problem in later releases.
> I'm not sure how I can use that to my advantage in 1.5.
>
> Incidentally, someone here did suggest re-implementing the whole
> threadpool (ExecutorService) ourselves to "work around" the issue.  I'll
> consider that notion if the casts and the isCancelled() call show up in
> profiling, but for now I'd like to stick with the standard libraries if
> possible.
>
> Regards,
> John.
>
> -----Original Message-----
> From: Joe Bowbeer [mailto:joe.bowbeer at gmail.com]
> Sent: Tuesday, October 25, 2005 1:24 PM
> To: Adams, John [CAR:3P41:EXCH]
> Cc: concurrency-interest at altair.cs.oswego.edu
> Subject: Re: [concurrency-interest] Exception handling & Executors
>
>
> A few ideas:
>
> 1. If you execute Runnable instead of FutureTask then any
> RuntimeExceptions and Errors thrown can escape the Executor.
>
> 2. You can hook the ThreadPoolExecutor.afterExecute method to handle the
> RuntimeExceptions thrown by your task.  (Note: the Throwable signature
> is a little misleading.)
>
> 3. You can use the Thread.set[Default]UncaughtExceptionHandler to handle
> uncaught exceptions and errors of all types.  (To associate the
> exception handling with your executor, you can provide your own thread
> factory.)
>
> 4. If you want to use FutureTask then you can hook its done() method and
> deal with the exception there.
>
>
> A comment:
>
> I don't know how you expect to do anything with an OutOfMemoryException,
> except perhaps log it.  I don't know of any way to reliably recover from
> these.
>
>
> On 10/25/05, John Adams <jadams3 at nortel.com> wrote:
> >
> > Hello,
> >
> > First off, thanks for maintaining this list, it has proven invaluable
> > to me as I learn to use the concurrency utilities in Java 5.0.
> >
> > The reason I'm posting is that I've had a lot of problems using the
> > Executor framework to deal with exceptions from errant tasks.  I've
> > complained about this to Sun, whose response was essentially that
> > exceptions belong with the task, so they should be try / caught at
> > that level.
> >
> > I don't agree.  I don't, for example, believe that it is the
> > responsibility of the task to handle OutOfMemoryError's, or in fact
> most of the classes
> > that occur within the Error class of throwables.   Many of those
> exceptions
> > imply that the tasks should have some knowledge about how to respond
> > to events are within the domain of the platform the tasks are
> > executing on. Now you could reasonably argue --> What is there to do
> anyways in that case
> > ?   All I want to do is a System.exit(), since we have a watchdog
> process
> > that does "the right thing" when the VM goes down.  In theory that
> > would have been easy for me to do with our custom threadpools pre-1.5,
>
> > however to get it working with the concurrent utilities has been a
> > painful experience.
> >
> > I thought it may be helpful to note what we tried in what order, so
> > that maybe we could get some support beyond the custom Future task
> > support going in later VM releases.  So, in order ....
> >
> > 1. Override the ThreadGroup before creating the Executor.    We did
> this
> > initially expecting things to be easy, however we were quickly
> > thwarted by the DefaultThreadFactory that does this ...
> >
> >        DefaultThreadFactory() {
> >             SecurityManager s = System.getSecurityManager();
> >             group = (s != null)? s.getThreadGroup() :
> >
> > Thread.currentThread().getThreadGroup();
> >          }
> >
> > I can see some security reasons for this, however I was not expecting
> > it --> I have stages that I wanted to log & name according to
> > threadgroups. However, that's not insurmountable ... I created my own
> > threadfactory which just took the currentThread().getThreadGroup() ...
> no luck.
> >   I also used the setExceptionHandler() method on Thread &
> > ThreadGroup, but to no avail.  I left this code in since I wanted this
>
> > code to be independent of our security manager choices, and explicitly
>
> > setting the exception handler seemed like a reasonable thing to do.
> > (and maybe relevant to ppl on this list)
> >
> > 2.  So then we went to look at the Future and realized what was
> > happening, the future task was catching throwable. (Yes, I know all
> > about custom futures, they don't help me now)  The only way we could
> > rationally seem to do this is to override afterExecute() in the
> > executor and do the following ....
> >
> > try {
> >                 // note I'd rather cast than use temporary objects
> >                 if ( !((FutureTask) task).isCancelled()){  // I don't
> > care about CancellationExceptions
> >                     ((FutureTask) task).get();
> >                 }
> >              } catch (ExecutionException ee) {
> >                  // ie if (ee == OutOfMemoryError) {System.exit()}
> >             }
> > }
> >
> > 3. Ok, so now for every task we're checking if it is cancelled,
> > casting 2x, and getting a result we almost never want, all of which
> have a performance
> > impact.   I tried throwing an OutOfMemoryError .... didn't trip the
> exit().
> > More than a little aggravated, I looked at the exception, and realized
>
> > that ExecutionException wraps Throwable, which means I need to inspect
>
> > the cause.  So I changed that bit of code to
> >
> >
> > try {
> >                if ( !((FutureTask) task).isCancelled()){ // I still
> > don't care about CancellationExceptions
> >                     ((FutureTask) task).get();
> >                 }
> >              } catch (ExecutionException ee) {
> >                  //  ie if (ee.getCause() == OutOfMemoryError)
> > {System.exit()}
> >             }
> > }
> >
> > 4.  The fourth problem I had was a bit subtle, and didn't come up
> until
> > later.   We have subsequent tasks that queue up on other executors.
> This
> > was resulting in a chain of ExecutionExceptions being set up, which
> > meant we had to recursively inspect all exceptions in the chain to
> make sure there
> > was no out of memory error.   Think SEDA, and you'll be able to
> picture why
> > we have multiple stages and chained execution exceptions.  It also
> > makes us more vulnerable to StackOverflow errors.
> >
> > Once you pull that all together, you can catch the OutOfMemoryError
> > cleanly, I think.  The moral of this story may be to not use the
> > Executors if you want to be a framework and don't trust your tasks,
> > but I don't think it has to be this hard.  There are several places in
>
> > the Java, and patterns that I can think of that have a
> > setExceptionHandler() pattern going on, wouldn't it be reasonable to
> > add similar functionality instead of forcing us to go through these
> > hoops to find an Error ?
> >
> > Again, all of this can be avoided if you trust your tasks to do a try
> > / catch (Throwable t) , but as part of a framework I don't exactly
> > have that luxury.  Thanks again for all the information, I hope this
> > is useful to others on this list.
> >
> > Regards,
> > John Adams
> >
>
>



More information about the Concurrency-interest mailing list