[concurrency-interest] Joint Talk for JavaOne?

Gregg Wonderly gregg at cytetech.com
Mon Apr 9 09:14:59 EDT 2012


Ahh, but this is part of API design.  You need to be able to "predict" execution 
paths by controlling them.  Complex state changes inside complex objects need to 
be controlled by a state machine, not by external "action order".  There are 
lots of reasons why good software design education, and training/experience is 
always a good thing.  Tools help when you are stuck with unknown, or 
inaccessible source bases.

For me, a stack trace is always what I use.  When any software I am supporting 
is behaving unpredictably, I will have already done several things, when putting 
it into operation.

1. I will have started it as a service on a UNIX like OS, so that I can see into 
it with many different tools, including:

	o lsof - to see what files are open to what destinations
	o /proc - or other process meta-data to see number of open files,
           memory use etc.
	o stdout and stderr will be redirected to append to a file at startup,
           so that I can just do "ps" to find it, and "kill -3" to see where
           threads are stuck at. I may even do kill -11 on it so that it stops
           with a JDK fault analysis which will expose library versions and
           several other internal things.

The script that runs the service will look like

while :
do
	echo "----------------- ${PRODUCT} Starting ------------------" >>$log
	${JAVA} ${opts} ${PRODCLASS} ${args} >>$log 2>&1 </dev/null &
         pid=$!
	echo $pid >${PRODHOME}/${PRODUCT}.pid
	# make killing this script kill the service
	trap "kill $pid;exit " 0 1 15
	wait
	# process exited without trapped signal, restart it
	trap - 0 1 15
	echo "----------  ${PRODUCT} Shutdown  ----------------------------------" >>$log
	echo "---------------- `date` -----------------" >>$log
	echo "---------------------------------------------------------------" >>$log
	# If still running, another was started, exit to avoid
	# creating lots
	if ps -awwwx | grep $JAVA | grep ${PRODCLASS}; then
		echo "$0: another instance of ${PROD} was found running, exiting" >&2
		exit
	fi
done

Setting around for many minutes, with a SSH session tailing $log, and another 
hitting it occasionally with kill -3, provides a tremendous amount of deadlock 
detection, but also it shows you hot spots.  You can see threads stuck waiting 
on locks (which aren't deadlock sources) and know that perhaps that point of 
contention is hotter than you imagined, or wanted it to be.

But, in the end, API design should really control paths so that you can, 
internally optimize the performance through those paths without having to 
'change' a bunch of code, refactoring access to make some contention or deadlock 
go away.

Gregg Wonderly

On 4/9/2012 7:32 AM, Kirk Pepperdine wrote:
> since it's very difficult to predict execution paths and execution paths are the
> source of the problem.....
> On 2012-04-09, at 1:24 PM, Ioannis Mavroukakis wrote:
>
>> Hello everyone,
>>
>>
>> Although I will not be attending JavaOne, I would like to offer my opinion.
>> Automatic detection is a nice to have but I believe that as a developer, I
>> would like to be educated in the "eyball" ways that exist to detect an issue.
>> Although one would argue (and I believe strongly in that) in "prevent, don't
>> cure" , brown runny stuff happens and something will creep past you. In that
>> instance, such a workshop by Dr Kabutz would be of high educational value.
>>
>>
>> Cheers,
>>
>> Ioannis
>>
>> On 9 April 2012 09:17, Kirk Pepperdine <kirk at kodewerk.com
>> <mailto:kirk at kodewerk.com>> wrote:
>>
>>     Hi William,
>>
>>     I'm on the core track committee and I vote based on 1) content (based on
>>     the strength of the abstract and supporting materials), 2) the % of
>>     informercial content (lower is better IMHO) 3) my feeling on the speakers
>>     ability to deliver. Heinz's doesn't get a pass because he's a very good
>>     friend. He'll get's my vote based on the criteria laid out. But since I've
>>     seen many of Heinz's talks I know that he'll pass on points 2 and 3 and I
>>     doubt he'll fail on point 1. The risk is, he's going for a hands on lab
>>     and from my experience in the past, he's going to need an extra helping
>>     hand from inside Oracle to get that approved. So, if you've a talk to
>>     submit, submit it and I promise it will be evaluated on it's merits. But
>>     do remember, there are a large number of fantastic abstracts on the
>>     cutting room floor just because our slot allocated wasn't enough to take
>>     them all.
>>
>>     Kirk
>>
>>     On 2012-04-09, at 8:45 AM, William Louth (JINSPIRED.COM
>>     <http://JINSPIRED.COM/>) wrote:
>>
>>>     I'd much prefer a session that discussed ways to automatically detect
>>>     such issues before they happened (at development and more importantly in
>>>     production) as well as runtime mechanisms (control/scheduling actions) at
>>>     such point to avoid them...I don't know anyone that wants to find such
>>>     issues in production when they only course of action is too simply kill
>>>     the process (though the cloud or other more fault "tolerant" software
>>>     makes this slightly less discomforting for some assume no residue
>>>     following on). In terms of avoidance I am assuming that the developer
>>>     can't simply rewrite everything from scratch to be align to some form of
>>>     packet messaging.
>>>
>>>     btw the way does the JavaOne conf still have rules on canvasing amongst
>>>     those involved in the selection of talks?
>>>
>>>     On 09/04/2012 08:08, Dr Heinz M. Kabutz wrote:
>>>>     15.000 lines is a tiny code base. The problem is very easy to solve once
>>>>     you get past the initial red herring. It took Nathan Reynolds two
>>>>     minutes to discover and fix it. If you look at the stack traces, there
>>>>     is only one possible place it could be and the stack trace takes you
>>>>     straight to the class and line number where the problem is.
>>>>
>>>>     That said, I would be very interested to see if there are any tools that
>>>>     could discover this deadlock statically or dynamically.
>>>>     Regards
>>>>
>>>>     Heinz
>>>>     --
>>>>     Dr Heinz M. Kabutz (PhD CompSci)
>>>>     Author of "The Java(tm) Specialists' Newsletter"
>>>>     Sun Java Champion
>>>>     IEEE Certified Software Development Professional
>>>>     http://www.javaspecialists.eu  <http://www.javaspecialists.eu/>
>>>>     Tel:+30 69 75 595 262  <tel:%2B30%2069%2075%20595%20262>
>>>>     Skype: kabutz
>>>>
>>>>
>>>>     On 4/9/12 6:23 AM, Mohan Radhakrishnan wrote:
>>>>>     I could look at the first deadlock by dumping locks and threads. I was
>>>>>     planning to find the next one using Java Path Finder but I think it is
>>>>>     not recommended for such a big codebase. I am not sure. Its heuristics
>>>>>     might take too much memory.
>>>>>
>>>>>     Mohan
>>>>>
>>>>>     On Mon, Apr 9, 2012 at 3:59 AM, <mabrouk2005 at gmail.com
>>>>>     <mailto:mabrouk2005 at gmail.com>> wrote:
>>>>>
>>>>>         Nice exercise Dr Heinz,
>>>>>
>>>>>         As others mentioned there were 2 kind of deadlocks the first one was
>>>>>         very is easy to find via jconsole so likeI thought that was it and
>>>>>         went and fixed that but then when ran it again it hang it so I knew
>>>>>         thee was more. the interesting thing the jconsole did not detect the
>>>>>         AWT deadlock it showed no deadlock even though the app hang. I had to
>>>>>         force stack dump and get clues to where is the deadlock. Does anyone
>>>>>         have experienced something like this with jconsole not detecting AWT
>>>>>         deadlocks?
>>>>>
>>>>>         Thanks again for the nice exercise and the great java news letter I am
>>>>>         a big fan.
>>>>>
>>>>>         Cheers,
>>>>>         Mabrouk
>>>>>
>>>>>         On Sat, Apr 7, 2012 at 7:42 AM, Remi Forax <forax at univ-mlv.fr
>>>>>         <mailto:forax at univ-mlv.fr>> wrote:
>>>>>         > No, it's more an AWRY or Java2D bug.
>>>>>         >
>>>>>         > Rémi
>>>>>         >
>>>>>         > Sent from my Phone
>>>>>         >
>>>>>         >
>>>>>         > ----- Reply message -----
>>>>>         > From: "Ben Evans" <benjamin.john.evans at gmail.com
>>>>>         <mailto:benjamin.john.evans at gmail.com>>
>>>>>         > To: "Dr Heinz M. Kabutz" <heinz at javaspecialists.eu
>>>>>         <mailto:heinz at javaspecialists.eu>>
>>>>>         > Cc: "Rémi Forax" <forax at univ-mlv.fr <mailto:forax at univ-mlv.fr>>,
>>>>>         <concurrency-interest at cs.oswego.edu
>>>>>         <mailto:concurrency-interest at cs.oswego.edu>>
>>>>>         > Subject: [concurrency-interest] Joint Talk for JavaOne?
>>>>>         > Date: Sat, Apr 7, 2012 10:43
>>>>>         >
>>>>>         >
>>>>>         > Heinz, was this with the new jdk8 + lambda build?
>>>>>         >
>>>>>         > If so, we should report this crash back to lambda-dev.
>>>>>         >
>>>>>         > Thanks,
>>>>>         >
>>>>>         > Ben
>>>>>         >
>>>>>         > On Sat, Apr 7, 2012 at 11:17 AM, Dr Heinz M. Kabutz
>>>>>         > <heinz at javaspecialists.eu <mailto:heinz at javaspecialists.eu>> wrote:
>>>>>         >> I see JDK8 crashes even with the vanilla Java2Demo from JDK 1.6.
>>>>>         I see
>>>>>         >> that
>>>>>         >> the Java2Demo has been removed from the demos folder of JDK 1.7
>>>>>         and 1.8.
>>>>>         >> It
>>>>>         >> thus is a Java 8 issue, rather than the changes I made to the code.
>>>>>         >> Thanks
>>>>>         >> for the warning Rémi.
>>>>>         >>
>>>>>         >>
>>>>>         >> Regards
>>>>>         >>
>>>>>         >> Heinz
>>>>>         >> --
>>>>>         >> Dr Heinz M. Kabutz (PhD CompSci)
>>>>>         >> Author of "The Java(tm) Specialists' Newsletter"
>>>>>         >> Sun Java Champion
>>>>>         >> IEEE Certified Software Development Professional
>>>>>         >> http://www.javaspecialists.eu <http://www.javaspecialists.eu/>
>>>>>         >> Tel: +30 69 75 595 262 <tel:%2B30%2069%2075%20595%20262>
>>>>>         >> Skype: kabutz
>>>>>         >>
>>>>>         >>
>>>>>         >> On 4/6/12 1:51 AM, Rémi Forax wrote:
>>>>>         >>>
>>>>>         >>> jdk8 crashes if I click on images of the tapped pane.
>>>>>         >>>
>>>>>         >>> Rémi
>>>>>         >>>
>>>>>         >>> On 04/05/2012 10:35 PM, Dr Heinz M. Kabutz wrote:
>>>>>         >>>>
>>>>>         >>>> Yes, you are correct.
>>>>>         >>>>
>>>>>         >>>> However, the mistake most programmers make is to #1 scratch
>>>>>         around in
>>>>>         >>>> the
>>>>>         >>>> source code to try and discover the problem and #2 then fix it
>>>>>         without
>>>>>         >>>> really understanding what they are doing. I think the lesson
>>>>>         is bigger
>>>>>         >>>> if
>>>>>         >>>> we give them the source code. In addition, it is probably more
>>>>>         >>>> difficult to
>>>>>         >>>> solve if you have the sources to muse over.
>>>>>         >>>> Regards
>>>>>         >>>>
>>>>>         >>>> Heinz
>>>>>         >>>> --
>>>>>         >>>> Dr Heinz M. Kabutz (PhD CompSci)
>>>>>         >>>> Author of "The Java(tm) Specialists' Newsletter"
>>>>>         >>>> Sun Java Champion
>>>>>         >>>> IEEE Certified Software Development Professional
>>>>>         >>>> http://www.javaspecialists.eu <http://www.javaspecialists.eu/>
>>>>>         >>>> Tel: +30 69 75 595 262 <tel:%2B30%2069%2075%20595%20262>
>>>>>         >>>> Skype: kabutz
>>>>>         >>>>
>>>>>         >>>>
>>>>>         >>>> On 4/5/12 11:30 PM, Nathan Reynolds wrote:
>>>>>         >>>>>
>>>>>         >>>>> I guess it is a good idea to not initially include the source
>>>>>         code in
>>>>>         >>>>> the workshop because many times I initially don't have the
>>>>>         source code
>>>>>         >>>>> when
>>>>>         >>>>> trying to figure out performance or scalability problems.
>>>>>         >>>>>
>>>>>         >>>>> Nathan Reynolds
>>>>>         >>>>> <http://psr.us.oracle.com/wiki/index.php/User:Nathan_Reynolds> |
>>>>>         >>>>> Consulting
>>>>>         >>>>> Member of Technical Staff | 602.333.9091
>>>>>         >>>>> Oracle PSR Engineering <http://psr.us.oracle.com/> | Server
>>>>>         Technology
>>>>>         >>>>>
>>>>>         >>>>> On 4/5/2012 1:22 PM, Dr Heinz M. Kabutz wrote:
>>>>>         >>>>>>
>>>>>         >>>>>> Sorry Nathan, here is the zip file including the source code
>>>>>         an an Ant
>>>>>         >>>>>> script to build the code. Sorry for accidentally excluding
>>>>>         that from
>>>>>         >>>>>> the
>>>>>         >>>>>> file I sent earlier. The problem was between the chair and the
>>>>>         >>>>>> keyboard.
>>>>>         >>>>>> However, one should be able to pinpoint the deadlock without
>>>>>         seeing a
>>>>>         >>>>>> line
>>>>>         >>>>>> of code. At the workshop I presented in Spain last week,
>>>>>         most of the
>>>>>         >>>>>> programmers immediately started delving into the code before
>>>>>         they knew
>>>>>         >>>>>> where
>>>>>         >>>>>> to look.
>>>>>         >>>>>> Regards
>>>>>         >>>>>>
>>>>>         >>>>>> Heinz
>>>>>         >>>>>> --
>>>>>         >>>>>> Dr Heinz M. Kabutz (PhD CompSci)
>>>>>         >>>>>> Author of "The Java(tm) Specialists' Newsletter"
>>>>>         >>>>>> Sun Java Champion
>>>>>         >>>>>> IEEE Certified Software Development Professional
>>>>>         >>>>>> http://www.javaspecialists.eu <http://www.javaspecialists.eu/>
>>>>>         >>>>>> Tel: +30 69 75 595 262 <tel:%2B30%2069%2075%20595%20262>
>>>>>         >>>>>> Skype: kabutz
>>>>>         >>>>>>
>>>>>         >>>>>> On 4/5/12 10:29 PM, Nathan Reynolds wrote:
>>>>>         >>>>>>>
>>>>>         >>>>>>> I took the bait and tried out the demo on HotSpot 7 Update
>>>>>         3. It
>>>>>         >>>>>>> didn't deadlock for a minute or two. Then the application
>>>>>         froze. I
>>>>>         >>>>>>> then
>>>>>         >>>>>>> used jstack to dump the call stacks. At the end of the
>>>>>         report, it
>>>>>         >>>>>>> said
>>>>>         >>>>>>> "Found one Java-level deadlock". It then dumped out the
>>>>>         locks and
>>>>>         >>>>>>> threads
>>>>>         >>>>>>> involved in the deadlock. I never looked at the source
>>>>>         code... in
>>>>>         >>>>>>> fact it
>>>>>         >>>>>>> wasn't in the zip file. This took about 5-7 minutes from
>>>>>         starting
>>>>>         >>>>>>> the
>>>>>         >>>>>>> application to finding the deadlock in the call stacks.
>>>>>         >>>>>>>
>>>>>         >>>>>>> A 100 good Java developers? Really? I guess I can
>>>>>         understand. Most
>>>>>         >>>>>>> engineers don't understand locks. Most of the time I get
>>>>>         answers
>>>>>         >>>>>>> like one
>>>>>         >>>>>>> should put a lock around shared variables. This is accurate
>>>>>         but they
>>>>>         >>>>>>> can't
>>>>>         >>>>>>> really go any deeper.
>>>>>         >>>>>>>
>>>>>         >>>>>>> It sounds like the material of your presentation should be
>>>>>         very good.
>>>>>         >>>>>>> It covers things that the deadlock detection tool probably
>>>>>         can't
>>>>>         >>>>>>> figure
>>>>>         >>>>>>> out.
>>>>>         >>>>>>>
>>>>>         >>>>>>> So, it seems like you have everything figured out. Why do
>>>>>         you need
>>>>>         >>>>>>> someone else?
>>>>>         >>>>>>>
>>>>>         >>>>>>> Nathan Reynolds
>>>>>         >>>>>>>
>>>>>         <http://psr.us.oracle.com/wiki/index.php/User:Nathan_Reynolds> |
>>>>>         >>>>>>> Consulting
>>>>>         >>>>>>> Member of Technical Staff | 602.333.9091
>>>>>         >>>>>>> Oracle PSR Engineering <http://psr.us.oracle.com/> | Server
>>>>>         >>>>>>> Technology
>>>>>         >>>>>>>
>>>>>         >>>>>>> On 4/5/2012 12:03 PM, Dr Heinz M. Kabutz wrote:
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> Hi Nathan,
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> a stack trace will only show you a certain class of obvious
>>>>>         >>>>>>>> deadlocks
>>>>>         >>>>>>>> that are easy to find and solve. I remember solving deadlocks
>>>>>         >>>>>>>> /before/ we
>>>>>         >>>>>>>> had the brilliant deadlock detection tool in the
>>>>>         ThreadMXBean. Yes,
>>>>>         >>>>>>>> the
>>>>>         >>>>>>>> deadlock detection tool shows you /some/ deadlocks, but
>>>>>         definitely
>>>>>         >>>>>>>> not all
>>>>>         >>>>>>>> of them. I can think of a bunch that it would not find:
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> 1. Semaphores running out of permits
>>>>>         >>>>>>>> 2. ReadWriteLocks trying to upgrade from write lock to
>>>>>         read lock
>>>>>         >>>>>>>> 3. Resource deadlocks
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> And some others that I will not describe in detail, such
>>>>>         as some
>>>>>         >>>>>>>> nasty livelocks that we can cause which will hang up the
>>>>>         JVM :-)
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> I agree with Kirk - it should probably be a 3 hour
>>>>>         workshop. I've
>>>>>         >>>>>>>> done this three times now - twice in Spain and once via
>>>>>         webinar.
>>>>>         >>>>>>>> The
>>>>>         >>>>>>>> workshop would involve some lessons on deadlocks, what
>>>>>         causes them
>>>>>         >>>>>>>> and how
>>>>>         >>>>>>>> we can prevent them. Also an explanation as to why we see
>>>>>         them so
>>>>>         >>>>>>>> seldom in
>>>>>         >>>>>>>> real production. I've got material prepared for this
>>>>>         workshop. And
>>>>>         >>>>>>>> then a
>>>>>         >>>>>>>> practical part to the workshop where they can try to find
>>>>>         a deadlock
>>>>>         >>>>>>>> in a
>>>>>         >>>>>>>> body of code.
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> Attached is an example of the workshop. You can try find the
>>>>>         >>>>>>>> deadlock if you like. I've shown it to almost 100 good Java
>>>>>         >>>>>>>> developers and
>>>>>         >>>>>>>> so far, not one has found it by themselves. I would expect
>>>>>         that
>>>>>         >>>>>>>> they should
>>>>>         >>>>>>>> find it in 5 minutes. In our workshop they would obviously be
>>>>>         >>>>>>>> helped a bit
>>>>>         >>>>>>>> with hints, so that they can walk away with a good
>>>>>         experience. Use
>>>>>         >>>>>>>> anything
>>>>>         >>>>>>>> you like. Oscilloscope, jstack, jconsole, whatever you
>>>>>         like. Only
>>>>>         >>>>>>>> rule for
>>>>>         >>>>>>>> you guys on the "concurrency-interest" list is: you can
>>>>>         only look at
>>>>>         >>>>>>>> the
>>>>>         >>>>>>>> code once you've found the problem, otherwise it would be
>>>>>         too easy
>>>>>         >>>>>>>> for you
>>>>>         >>>>>>>> :-) Once you have found it, you can also fix the code and
>>>>>         make sure
>>>>>         >>>>>>>> that
>>>>>         >>>>>>>> the code then works correctly.
>>>>>         >>>>>>>> Regards
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> Heinz
>>>>>         >>>>>>>> --
>>>>>         >>>>>>>> Dr Heinz M. Kabutz (PhD CompSci)
>>>>>         >>>>>>>> Author of "The Java(tm) Specialists' Newsletter"
>>>>>         >>>>>>>> Sun Java Champion
>>>>>         >>>>>>>> IEEE Certified Software Development Professional
>>>>>         >>>>>>>> http://www.javaspecialists.eu <http://www.javaspecialists.eu/>
>>>>>         >>>>>>>> Tel: +30 69 75 595 262
>>>>>         >>>>>>>> Skype: kabutz
>>>>>         >>>>>>>>
>>>>>         >>>>>>>> On 4/5/12 8:01 PM, Nathan Reynolds wrote:
>>>>>         >>>>>>>>>
>>>>>         >>>>>>>>> I am curious as to the intended content of your presentation.
>>>>>         >>>>>>>>> Here's some ideas. I wondering what you are planing on doing.
>>>>>         >>>>>>>>>
>>>>>         >>>>>>>>> 1. Printing the call stacks with lock information using a
>>>>>         tool
>>>>>         >>>>>>>>> such as jstack
>>>>>         >>>>>>>>> 2. Press the "Detect Deadlock" button in JConsole
>>>>>         >>>>>>>>> 3. Dealing with Lock and Condition objects instead of
>>>>>         >>>>>>>>> synchronized blocks
>>>>>         >>>>>>>>> 4. Detecting live lock
>>>>>         >>>>>>>>> 5. Distributed deadlocks
>>>>>         >>>>>>>>>
>>>>>         >>>>>>>>> Nathan Reynolds
>>>>>         >>>>>>>>>
>>>>>         <http://psr.us.oracle.com/wiki/index.php/User:Nathan_Reynolds> |
>>>>>         >>>>>>>>> Consulting
>>>>>         >>>>>>>>> Member of Technical Staff | 602.333.9091
>>>>>         >>>>>>>>> Oracle PSR Engineering <http://psr.us.oracle.com/> | Server
>>>>>         >>>>>>>>> Technology
>>>>>         >>>>>>>>>
>>>>>         >>>>>>>>> On 4/5/2012 7:57 AM, Dr Heinz M. Kabutz wrote:
>>>>>         >>>>>>>>>>
>>>>>         >>>>>>>>>> Good afternoon,
>>>>>         >>>>>>>>>>
>>>>>         >>>>>>>>>> I am thinking of submitting a talk on finding and diagnosing
>>>>>         >>>>>>>>>> deadlocks for JavaOne. It might be a workshop, instead of a
>>>>>         >>>>>>>>>> simple
>>>>>         >>>>>>>>>> conference talk. Or we might be able to turn the talk into a
>>>>>         >>>>>>>>>> workshop where
>>>>>         >>>>>>>>>> they need to find a deadlock in a body of code. Could be
>>>>>         >>>>>>>>>> something
>>>>>         >>>>>>>>>> different, which might go incredibly well or crash in a
>>>>>         blaze of
>>>>>         >>>>>>>>>> flames.
>>>>>         >>>>>>>>>> Would any of you be interested in co-presenting this
>>>>>         with me at
>>>>>         >>>>>>>>>> JavaOne?
>>>>>         >>>>>>>>>>
>>>>>         >>>>>>>>>> Regards
>>>>>         >>>>>>>>>>
>>>>>         >>>>>>>>>> Heinz
>>>>>         >>>>
>>>>>         >>>>
>>>>>         >>>>
>>>>>         >>>> _______________________________________________
>>>>>         >>>> Concurrency-interest mailing list
>>>>>         >>>> Concurrency-interest at cs.oswego.edu
>>>>>         <mailto:Concurrency-interest at cs.oswego.edu>
>>>>>         >>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>>         >>>
>>>>>         >>>
>>>>>         >>> _______________________________________________
>>>>>         >>> Concurrency-interest mailing list
>>>>>         >>> Concurrency-interest at cs.oswego.edu
>>>>>         <mailto:Concurrency-interest at cs.oswego.edu>
>>>>>         >>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>>         >>>
>>>>>         >> _______________________________________________
>>>>>         >> Concurrency-interest mailing list
>>>>>         >> Concurrency-interest at cs.oswego.edu
>>>>>         <mailto:Concurrency-interest at cs.oswego.edu>
>>>>>         >> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>>         >
>>>>>         > _______________________________________________
>>>>>         > Concurrency-interest mailing list
>>>>>         > Concurrency-interest at cs.oswego.edu
>>>>>         <mailto:Concurrency-interest at cs.oswego.edu>
>>>>>         > http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>>         >
>>>>>
>>>>>         _______________________________________________
>>>>>         Concurrency-interest mailing list
>>>>>         Concurrency-interest at cs.oswego.edu
>>>>>         <mailto:Concurrency-interest at cs.oswego.edu>
>>>>>         http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>>
>>>>>
>>>>>     --------------------------------------------------------------------------------
>>>>>
>>>>>     _______________________________________________
>>>>>     Concurrency-interest mailing list
>>>>>     Concurrency-interest at cs.oswego.edu  <mailto:Concurrency-interest at cs.oswego.edu>
>>>>>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>>
>>>>
>>>>
>>>>     _______________________________________________
>>>>     Concurrency-interest mailing list
>>>>     Concurrency-interest at cs.oswego.edu  <mailto:Concurrency-interest at cs.oswego.edu>
>>>>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>     _______________________________________________
>>>     Concurrency-interest mailing list
>>>     Concurrency-interest at cs.oswego.edu
>>>     <mailto:Concurrency-interest at cs.oswego.edu>
>>>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>
>>
>>     _______________________________________________
>>     Concurrency-interest mailing list
>>     Concurrency-interest at cs.oswego.edu <mailto:Concurrency-interest at cs.oswego.edu>
>>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>
>>
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu <mailto:Concurrency-interest at cs.oswego.edu>
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>
>
>
> _______________________________________________
> Concurrency-interest mailing list
> Concurrency-interest at cs.oswego.edu
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest



More information about the Concurrency-interest mailing list