[concurrency-interest] We need your help testing

Doug Lea dl@cs.oswego.edu
Fri, 20 Jun 2003 08:58:02 -0400

One of the next major tasks for JSR-166 is to provide a test suite for
API compliance (a TCK). To keep on JDK1.5 schedule, an initial version
of test suite is due by the end of July (i.e. SOON!).  Given the usual
problems in adequately testing concurrent code, we'd like to provide
something much better than the required minimal compliance tests, and
to include as diverse a set of tests as possible. The best way to get
diversity is to ask people using this code to help contribute. If you
have used dl.util.concurrent and/or anticipate using
java.util.concurrent, now is the best time to help ensure that you and
others can continue doing so in JDK1.5.

Among the easiest and best ways to help would be to take an existing
usage/application of these classes, and reduce it to something with a
yes/no result (or terminates-soon vs never-terminates).  If the tests
use dl.util.concurrent, we can convert to use new JSR-166 versions.
If you are using junit (http://junit.org), that would be great,
otherwise we can junit-ize or otherwise accommodate.

We'd like more tests of all kinds, from the simple to the esoteric.
These include:

* "Normative" tests. The include checks for whether documented
argument screening actually occurs and the expected exceptions are
thrown. For example, whether trying to make an ArrayBlockingQueue with
negative capacity fails in the specified way.

* Postcondition tests. Unit tests usually amounting to checks that
expected observable state changes in objects occur. For a simple
example does Cancellable.cancel cause isCancelled to be true.

* Liveness tests. Tests for common concurrency errors causing threads
to hang due to livelock, lockout, deadlock, missed signals, etc.  For
a simple example, a test in which many threads each lock and unlock a
number of ReentrantLocks exactly once should terminate.

* Race tests. Tests for insufficient synchronization, sometimes causing
bad field values to be accessed. For example, a test in which
a pair of values must always bear some known relationship, tested
with many threads continually reading and writing them.

* Performance anomaly tests. Tests that documented usage guidance
holds. For example, is LinkedBlockingQueue indeed on average
faster but with higher variance than ArrayBlockingQueue.

* Cancellation tests. Stopping actions (via "cancel" methods,
timeouts, interrupts, exceptions, JVM shutdown etc) cleanly and
correctly is among the hardest things to get right in concurrent
programming, and tends to hit the trickiest code. It is also difficult
to test for.  Any contrbutions aong these lines are especially

* Feature interaction tests. For example, do the tuning methods in
ThreadPoolExecutor combine correctly? "Correctly" here is not always
spelled out well. One purpose of such tests is to help clarify API
specs about these issues.  (Aside: There were a few such feature
interaction problems in dl.u.c.PooledExecutor, for example, that
resetting keepAlive times could delay shutdown. I think these are
cleared up in j.u.c version.)

* Unexpected usage tests. Functionality or performance problems when
classes are used in unanticipated ways or applications. At this point,
lots of usages are unanticipated, so almost anything fits this

* Stress tests.  Tests that check for performance degradation and
instability over long periods.

* Platform tests. Ensuring that documented properties hold
across uniprocessors and multiprocessors of various kinds.

* Least-astonishment tests. Intrinsically subjective tests that a
class/method does what it says it should do, but this isn't what you
think it should do. Plus related human-factors tests revealing awkward
or error-prone constructions that could be fixed with API changes.

* Code reviews. Walkthroughs of code looking for concurrency problems.
(See also a follow-up posting on this coming soon.)

* Code analysis. Analytic tools looking for concurrency problems.
(Note: This is even harder to do for JSR-166 code than it might be
otherwise because of the extensive use of from-the-ground-up
synch primitives, that are not amenable to checks based on the
use of "synchronized" methods. The goal of exposing ReentrantLock,
Atomics, etc., to as much internal reuse (hence internal testing) as
possible overrides ease-of-analysis concerns.)

We will of course make all appropriate tests freely avalaible as part
of the JSR-166 TCK. (If you have problems revealing code, and cannot
allow it to appear, we can probably work out something.) We'd be
very pleased to somehow give you credit for any contributions.

You can just send them directly to me <dl@cs.oswego.edu>. 
We'll figure out how to coordinate as needs arises.

Thanks very much for any help you can provide!

Doug Lea, Computer Science Department, SUNY Oswego, Oswego, NY 13126 USA
dl@cs.oswego.edu 315-312-2688 FAX:315-312-5424 http://gee.cs.oswego.edu/