<div dir="auto">There's the possibility that a task which unlocks the lock is stuck in the executor's submission queue because all the workers are blocked on obtaining the lock.<br><br><div data-smartmail="gmail_signature">-- <br>Cheers,<br>√</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Jun 17, 2017 8:13 AM, "Kirk Pepperdine" <<a href="mailto:kirk@kodewerk.com">kirk@kodewerk.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">There are 94 threads that appear to be waiting on 2 different locks. 47 are waiting on a ForkJoinExecutor and are simply idle. From this one would hope that you were on a 48 core machine.<div><br></div><div>As an example….</div><div><pre><font size="-1">"marathon-akka.actor.default-<wbr>dispatcher-35" #271 prio=5 os_prio=0 tid=0x00007ff868004000 nid=0x11fd4 runnable [0x00007ff8a9be0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <a><0x00000005c0581aa0></a> (a akka.dispatch.<wbr>ForkJoinExecutorConfigurator$<wbr>AkkaForkJoinPool)
        at scala.concurrent.forkjoin.<wbr>ForkJoinPool.scan(<wbr>ForkJoinPool.java:2075)
        at scala.concurrent.forkjoin.<wbr>ForkJoinPool.runWorker(<wbr>ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.<wbr>ForkJoinWorkerThread.run(<wbr>ForkJoinWorkerThread.java:107)</font></pre><div><br></div><div>26 are waiting on a ReentantLock. These are probably the threads of interest.</div><div><br></div><div><pre><font size="-1">"ForkJoinPool-2-worker-33" #25346 daemon prio=5 os_prio=0 tid=0x00007ff80c11c000 nid=0x5747 waiting on condition [0x00007ff8a91d8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <a><0x00000005c0af6298></a> (a java.util.concurrent.locks.<wbr>ReentrantLock$FairSync)
        at java.util.concurrent.locks.<wbr>LockSupport.park(LockSupport.<wbr>java:175)
        at java.util.concurrent.locks.<wbr>AbstractQueuedSynchronizer.<wbr>parkAndCheckInterrupt(<wbr>AbstractQueuedSynchronizer.<wbr>java:836)
        at java.util.concurrent.locks.<wbr>AbstractQueuedSynchronizer.<wbr>acquireQueued(<wbr>AbstractQueuedSynchronizer.<wbr>java:870)
        at java.util.concurrent.locks.<wbr>AbstractQueuedSynchronizer.<wbr>acquire(<wbr>AbstractQueuedSynchronizer.<wbr>java:1199)
        at java.util.concurrent.locks.<wbr>ReentrantLock$FairSync.lock(<wbr>ReentrantLock.java:224)
        at java.util.concurrent.locks.<wbr>ReentrantLock.lock(<wbr>ReentrantLock.java:285)
        at mesosphere.marathon.util.<wbr>RichLock$.apply$extension(<wbr>Lock.scala:7)
        at mesosphere.marathon.util.Lock.<wbr>apply(Lock.scala:25)
        at mesosphere.marathon.util.<wbr>KeyedLock.apply(WorkQueue.<wbr>scala:94)</font></pre><pre><font size="-1"><br></font></pre><pre><font face="Helvetica"><span style="white-space:normal">Are you waiting on an external signal?</span></font></pre><pre><font face="Helvetica"><span style="white-space:normal"><br></span></font></pre><pre><font face="Helvetica"><span style="white-space:normal">— Kirk</span></font></pre></div><div><br></div><div><br></div><div><br></div><div><br></div><div><div><blockquote type="cite"><div>On Jun 17, 2017, at 6:03 AM, Henri Tremblay <<a href="mailto:henri.tremblay@gmail.com" target="_blank">henri.tremblay@gmail.com</a>> wrote:</div><br class="m_-4048312078916616848Apple-interchange-newline"><div><div dir="ltr">If you keep doing thread dumps, is the lock and the waiting threads the same? So a real deadlock. If yes, Kirk suggestion is interesting. And I would also do a heap dump to have a look at the threads holding a reference to the reentrant lock.</div><div class="gmail_extra"><br><div class="gmail_quote">On 16 June 2017 at 18:55, Kirk Pepperdine <span dir="ltr"><<a href="mailto:kirk@kodewerk.com" target="_blank">kirk@kodewerk.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi Ken,<div><br></div><div>If you have a lock with nothing holding on to it then either a VM thread has the lock. The most common culprit are GC threads. I’ve not taken a deep look at the code but you could set -XX:+PrintConcurrentLocks which would give you an idea if the problem is with the re-enterant lock. Not looked too deeply at the code but I don’t see anything obvious at the moment.</div><div><br></div><div>— Kirk</div><div><br></div><div><div><blockquote type="cite"><div><div class="m_-4048312078916616848h5"><div>On Jun 16, 2017, at 11:36 PM, Ken Sipe <<a href="mailto:kensipe@gmail.com" target="_blank">kensipe@gmail.com</a>> wrote:</div><br class="m_-4048312078916616848m_-4626314978813734132Apple-interchange-newline"></div></div><div><div><div class="m_-4048312078916616848h5"><div style="word-wrap:break-word"><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,'Segoe UI',Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;font-variant-ligatures:normal;background-color:rgb(255,255,255)">We have encountered a strange deadlock scenario in which it appears that all threads are waiting on acquiring a ReentrantLock, but no thread has the lock. </span><div><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,'Segoe UI',Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;font-variant-ligatures:normal;background-color:rgb(255,255,255)"><br></span></div><div><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,'Segoe UI',Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;font-variant-ligatures:normal;background-color:rgb(255,255,255)">Any thoughts on this?</span></div><div><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,'Segoe UI',Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;font-variant-ligatures:normal;background-color:rgb(255,255,255)"><br></span></div><div><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,'Segoe UI',Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;font-variant-ligatures:normal;background-color:rgb(255,255,255)"><a href="https://gist.github.com/timcharper/9ab4fea9da4669e620507e85e764d94a" target="_blank">https://gist.github.com/timcha<wbr>rper/9ab4fea9da4669e620507e85e<wbr>764d94a</a></span></div><div><br></div><div><br></div><div>Thanks,</div><div>ken</div><div><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,'Segoe UI',Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;font-variant-ligatures:normal;background-color:rgb(255,255,255)"><br></span></div><div><span style="color:rgb(36,41,46);font-family:-apple-system,system-ui,'Segoe UI',Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;font-variant-ligatures:normal;background-color:rgb(255,255,255)"><br></span></div></div></div></div>______________________________<wbr>_________________<br>Concurrency-interest mailing list<br><a href="mailto:Concurrency-interest@cs.oswego.edu" target="_blank">Concurrency-interest@cs.oswego<wbr>.edu</a><br><a href="http://cs.oswego.edu/mailman/listinfo/concurrency-interest" target="_blank">http://cs.oswego.edu/mailman/l<wbr>istinfo/concurrency-interest</a><br></div></blockquote></div><br></div></div><br>______________________________<wbr>_________________<br>
Concurrency-interest mailing list<br>
<a href="mailto:Concurrency-interest@cs.oswego.edu" target="_blank">Concurrency-interest@cs.oswego<wbr>.edu</a><br>
<a href="http://cs.oswego.edu/mailman/listinfo/concurrency-interest" rel="noreferrer" target="_blank">http://cs.oswego.edu/mailman/l<wbr>istinfo/concurrency-interest</a><br>
<br></blockquote></div><br></div>
</div></blockquote></div><br></div></div></div><br>______________________________<wbr>_________________<br>
Concurrency-interest mailing list<br>
<a href="mailto:Concurrency-interest@cs.oswego.edu">Concurrency-interest@cs.<wbr>oswego.edu</a><br>
<a href="http://cs.oswego.edu/mailman/listinfo/concurrency-interest" rel="noreferrer" target="_blank">http://cs.oswego.edu/mailman/<wbr>listinfo/concurrency-interest</a><br>
<br></blockquote></div></div>