[concurrency-interest] Exception handling & Executors

John Adams jadams3 at nortel.com
Tue Oct 25 15:59:24 EDT 2005


Hi Joe,

Thank you for the reply.  Regarding your comment about OutOfMemory, as
was buried in the original post, I want to log the exception and then do
a System.exit().  We have a watchdog process that restarts the server
and starts again.  If we have a gradual leak going on, it's better in
the field for us to raise an alarm and restart the server than it is to
thrash with out of memory.  We're trying to build a reliable system, and
we have to deal with the possibility that this could happen in the
field, despite all of our best leak detection efforts.  

To respond ...

1. We use runnables in our implementation already.  If you pass a
runnable into a task, this code in AbstractExecutorService gets hit ...

 public <T> Future<T> submit(Runnable task, T result) {
        if (task == null) throw new NullPointerException();
        FutureTask<T> ftask = new FutureTask<T>(task, result);
        execute(ftask);
        return ftask;
    }

Note the wrapping of the FutureTask, and the source of my complaint.
The inner task uses Sync, an inner class of FutureTask, which does this
..

 void innerRun() {
            if (!compareAndSetState(0, RUNNING)) 
                return;
            try {
                runner = Thread.currentThread();
                innerSet(callable.call());
            } catch(Throwable ex) {
                innerSetException(ex);
            } 
        }

2. I do hook into the afterExecute() which resolves my problem, albeit
with some ugly code.   I'm really just saying that this solution is
complex and hoping someone in the future will implement a
setDefaultExceptionHandler() equivalent for the Executor framework.  I
also thought others might have this problem, so it would be useful to
provide this code to others with similar requirements.

3. The Thread.setDefaultExceptionHandler doesn't resolve the problem,
the FutureTask/Sync catches throwable. 

4. The done() idea is interesting, although I'm guessing that's part of
how the custom future task will deal with the problem in later releases.
I'm not sure how I can use that to my advantage in 1.5.

Incidentally, someone here did suggest re-implementing the whole
threadpool (ExecutorService) ourselves to "work around" the issue.  I'll
consider that notion if the casts and the isCancelled() call show up in
profiling, but for now I'd like to stick with the standard libraries if
possible. 

Regards,
John.

-----Original Message-----
From: Joe Bowbeer [mailto:joe.bowbeer at gmail.com] 
Sent: Tuesday, October 25, 2005 1:24 PM
To: Adams, John [CAR:3P41:EXCH]
Cc: concurrency-interest at altair.cs.oswego.edu
Subject: Re: [concurrency-interest] Exception handling & Executors


A few ideas:

1. If you execute Runnable instead of FutureTask then any
RuntimeExceptions and Errors thrown can escape the Executor.

2. You can hook the ThreadPoolExecutor.afterExecute method to handle the
RuntimeExceptions thrown by your task.  (Note: the Throwable signature
is a little misleading.)

3. You can use the Thread.set[Default]UncaughtExceptionHandler to handle
uncaught exceptions and errors of all types.  (To associate the
exception handling with your executor, you can provide your own thread
factory.)

4. If you want to use FutureTask then you can hook its done() method and
deal with the exception there.


A comment:

I don't know how you expect to do anything with an OutOfMemoryException,
except perhaps log it.  I don't know of any way to reliably recover from
these.


On 10/25/05, John Adams <jadams3 at nortel.com> wrote:
>
> Hello,
>
> First off, thanks for maintaining this list, it has proven invaluable 
> to me as I learn to use the concurrency utilities in Java 5.0.
>
> The reason I'm posting is that I've had a lot of problems using the 
> Executor framework to deal with exceptions from errant tasks.  I've 
> complained about this to Sun, whose response was essentially that 
> exceptions belong with the task, so they should be try / caught at 
> that level.
>
> I don't agree.  I don't, for example, believe that it is the 
> responsibility of the task to handle OutOfMemoryError's, or in fact
most of the classes
> that occur within the Error class of throwables.   Many of those
exceptions
> imply that the tasks should have some knowledge about how to respond 
> to events are within the domain of the platform the tasks are 
> executing on. Now you could reasonably argue --> What is there to do
anyways in that case
> ?   All I want to do is a System.exit(), since we have a watchdog
process
> that does "the right thing" when the VM goes down.  In theory that 
> would have been easy for me to do with our custom threadpools pre-1.5,

> however to get it working with the concurrent utilities has been a 
> painful experience.
>
> I thought it may be helpful to note what we tried in what order, so 
> that maybe we could get some support beyond the custom Future task 
> support going in later VM releases.  So, in order ....
>
> 1. Override the ThreadGroup before creating the Executor.    We did
this
> initially expecting things to be easy, however we were quickly 
> thwarted by the DefaultThreadFactory that does this ...
>
>        DefaultThreadFactory() {
>             SecurityManager s = System.getSecurityManager();
>             group = (s != null)? s.getThreadGroup() :
>
> Thread.currentThread().getThreadGroup();
>          }
>
> I can see some security reasons for this, however I was not expecting 
> it --> I have stages that I wanted to log & name according to 
> threadgroups. However, that's not insurmountable ... I created my own 
> threadfactory which just took the currentThread().getThreadGroup() ...
no luck.
>   I also used the setExceptionHandler() method on Thread & 
> ThreadGroup, but to no avail.  I left this code in since I wanted this

> code to be independent of our security manager choices, and explicitly

> setting the exception handler seemed like a reasonable thing to do.  
> (and maybe relevant to ppl on this list)
>
> 2.  So then we went to look at the Future and realized what was 
> happening, the future task was catching throwable. (Yes, I know all 
> about custom futures, they don't help me now)  The only way we could 
> rationally seem to do this is to override afterExecute() in the 
> executor and do the following ....
>
> try {
>                 // note I'd rather cast than use temporary objects
>                 if ( !((FutureTask) task).isCancelled()){  // I don't 
> care about CancellationExceptions
>                     ((FutureTask) task).get();
>                 }
>              } catch (ExecutionException ee) {
>                  // ie if (ee == OutOfMemoryError) {System.exit()}
>             }
> }
>
> 3. Ok, so now for every task we're checking if it is cancelled, 
> casting 2x, and getting a result we almost never want, all of which
have a performance
> impact.   I tried throwing an OutOfMemoryError .... didn't trip the
exit().
> More than a little aggravated, I looked at the exception, and realized

> that ExecutionException wraps Throwable, which means I need to inspect

> the cause.  So I changed that bit of code to
>
>
> try {
>                if ( !((FutureTask) task).isCancelled()){ // I still 
> don't care about CancellationExceptions
>                     ((FutureTask) task).get();
>                 }
>              } catch (ExecutionException ee) {
>                  //  ie if (ee.getCause() == OutOfMemoryError) 
> {System.exit()}
>             }
> }
>
> 4.  The fourth problem I had was a bit subtle, and didn't come up
until
> later.   We have subsequent tasks that queue up on other executors.
This
> was resulting in a chain of ExecutionExceptions being set up, which 
> meant we had to recursively inspect all exceptions in the chain to
make sure there
> was no out of memory error.   Think SEDA, and you'll be able to
picture why
> we have multiple stages and chained execution exceptions.  It also 
> makes us more vulnerable to StackOverflow errors.
>
> Once you pull that all together, you can catch the OutOfMemoryError 
> cleanly, I think.  The moral of this story may be to not use the 
> Executors if you want to be a framework and don't trust your tasks, 
> but I don't think it has to be this hard.  There are several places in

> the Java, and patterns that I can think of that have a 
> setExceptionHandler() pattern going on, wouldn't it be reasonable to 
> add similar functionality instead of forcing us to go through these 
> hoops to find an Error ?
>
> Again, all of this can be avoided if you trust your tasks to do a try 
> / catch (Throwable t) , but as part of a framework I don't exactly 
> have that luxury.  Thanks again for all the information, I hope this 
> is useful to others on this list.
>
> Regards,
> John Adams
>




More information about the Concurrency-interest mailing list