[concurrency-interest] Deadlock with when no threads have the lock

Kirk Pepperdine kirk at kodewerk.com
Sat Jun 17 02:12:18 EDT 2017


There are 94 threads that appear to be waiting on 2 different locks. 47 are waiting on a ForkJoinExecutor and are simply idle. From this one would hope that you were on a 48 core machine.

As an example….
"marathon-akka.actor.default-dispatcher-35" #271 prio=5 os_prio=0 tid=0x00007ff868004000 nid=0x11fd4 runnable [0x00007ff8a9be0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000005c0581aa0> <monitor://<0x00000005c0581aa0>> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

26 are waiting on a ReentantLock. These are probably the threads of interest.

"ForkJoinPool-2-worker-33" #25346 daemon prio=5 os_prio=0 tid=0x00007ff80c11c000 nid=0x5747 waiting on condition [0x00007ff8a91d8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000005c0af6298> <monitor://<0x00000005c0af6298>> (a java.util.concurrent.locks.ReentrantLock$FairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:224)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at mesosphere.marathon.util.RichLock$.apply$extension(Lock.scala:7)
	at mesosphere.marathon.util.Lock.apply(Lock.scala:25)
	at mesosphere.marathon.util.KeyedLock.apply(WorkQueue.scala:94)

Are you waiting on an external signal?

— Kirk




> On Jun 17, 2017, at 6:03 AM, Henri Tremblay <henri.tremblay at gmail.com> wrote:
> 
> If you keep doing thread dumps, is the lock and the waiting threads the same? So a real deadlock. If yes, Kirk suggestion is interesting. And I would also do a heap dump to have a look at the threads holding a reference to the reentrant lock.
> 
> On 16 June 2017 at 18:55, Kirk Pepperdine <kirk at kodewerk.com <mailto:kirk at kodewerk.com>> wrote:
> Hi Ken,
> 
> If you have a lock with nothing holding on to it then either a VM thread has the lock. The most common culprit are GC threads. I’ve not taken a deep look at the code but you could set -XX:+PrintConcurrentLocks which would give you an idea if the problem is with the re-enterant lock. Not looked too deeply at the code but I don’t see anything obvious at the moment.
> 
> — Kirk
> 
>> On Jun 16, 2017, at 11:36 PM, Ken Sipe <kensipe at gmail.com <mailto:kensipe at gmail.com>> wrote:
>> 
>> We have encountered a strange deadlock scenario in which it appears that all threads are waiting on acquiring a ReentrantLock, but no thread has the lock. 
>> 
>> Any thoughts on this?
>> 
>> https://gist.github.com/timcharper/9ab4fea9da4669e620507e85e764d94a <https://gist.github.com/timcharper/9ab4fea9da4669e620507e85e764d94a>
>> 
>> 
>> Thanks,
>> ken
>> 
>> 
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu <mailto:Concurrency-interest at cs.oswego.edu>
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest <http://cs.oswego.edu/mailman/listinfo/concurrency-interest>
> 
> 
> _______________________________________________
> Concurrency-interest mailing list
> Concurrency-interest at cs.oswego.edu <mailto:Concurrency-interest at cs.oswego.edu>
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest <http://cs.oswego.edu/mailman/listinfo/concurrency-interest>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20170617/afb4bed1/attachment-0001.html>


More information about the Concurrency-interest mailing list