[concurrency-interest] ClassLoader deadlock

Peter Levart peter.levart at gmail.com
Sat Feb 6 13:27:29 EST 2016



On 02/06/2016 01:17 PM, Peter Firmstone wrote:
> The security manager is non blocking, unfortunately java system 
> classes aren't.

It doesn't seem so. At least, I can't find where main thread locks the 
0x040ebee8 monitor:

"Thread-1":
   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
   which is held by "main"


"main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor 
entry [0x0185e000]
    java.lang.Thread.State: BLOCKED (on object monitor)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
         - waiting to lock <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at 
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
         at 
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at 
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at 
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at 
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
         at 
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
         at 
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
         at 
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at 
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
         at 
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
         at 
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
         at java.lang.System.checkIO(System.java:253)
         at java.lang.System.setErr(System.java:199)
         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)


...no mention of 0x040ebee8 in main thread...?!

Peter

>
> The policy provider in use thread confines PermissionCollection 
> instances, which never leave the cpu cache, they are discarded 
> immediately after use.
>
> The problem here is that two different threads are attempting to load 
> the same class at the same time.
>
> I'll have to make sure that all classes are loaded before the security 
> manager becomes the system security manager, either that or eliminate 
> the AccessControlContext check permission cache.
>
> While the standard java security manager and policy provider consume 
> around 10% cpu, this consumes less than 1%.
>
> On 6/02/2016 9:57 PM, David Holmes wrote:
>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>> deadlock prone?  Guess I was wrong.
>>
>> The deadlock is introduced by a custom security manager. SM's play a 
>> critical role in many parts of the core libraries so if they utilize 
>> synchronization then they can easily introduce deadlock.
>>
>>> Observation: On an unrelated occassion, I had a URLClassLoader
>>> synchronization hotspot (well not standard URLClassLoader, but a high
>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>> instead of URL DNS lookup), I solved that problem by thread confining
>>> class loading.  I've never experienced deadlock using thread
>>> confinement, initially I was worried that there would be
>>> interdependencies, however I didn't experience a problem, as the single
>>> thread would always load the classes it needed.
>>>
>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>> strategy, I found thread confinement was very performant.
>>
>> Are you suggesting that a class-load request is actually transmitted 
>> to a single class-loading thread, while the initial thread blocks 
>> until the loading is complete? That would still seem prone to the 
>> same SM related deadlock scenario. And it would not generally be 
>> particularly performant.
>
> I thought that too, until I tried it.
>
> Would you like to see some performance figures?
>
> Cheers,
>
> Peter.
>
>>
>> David
>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> 2016-02-06 21:06:07
>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>
>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>> runnable [0x00000000]
>>>     java.lang.Thread.State: RUNNABLE
>>>
>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>> nid=0x111c waiting on condition [0x00000000]
>>>     java.lang.Thread.State: RUNNABLE
>>>
>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
>>> waiting on condition [0x00000000]
>>>     java.lang.Thread.State: RUNNABLE
>>>
>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400 
>>> nid=0x10b8
>>> runnable [0x00000000]
>>>     java.lang.Thread.State: RUNNABLE
>>>
>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>> monitor entry [0x03ded000]
>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>          at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>          at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>
>>>
>>>          at
>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>          at
>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137) 
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070) 
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) 
>>>
>>>
>>>          at 
>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>          at 
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>          at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167) 
>>>
>>>
>>>          at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166) 
>>>
>>>
>>>          - locked <0x03ef8b30> (a
>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>          at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>          at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>          at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j 
>>>
>>>
>>> ava:180)
>>>          at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294 
>>>
>>>
>>> )
>>>          at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
>>>
>>>
>>>          at java.lang.Thread.run(Thread.java:744)
>>>
>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>> Object.wait() [0x03cff000]
>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          - waiting on <0x03e056d8> (a 
>>> java.lang.ref.ReferenceQueue$Lock)
>>>          at 
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>>          - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>          at 
>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>>          at 
>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>
>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>> nid=0x1398 in Object.wait() [0x144cf000]
>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>          at java.lang.Object.wait(Native Method)
>>>          - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>          at java.lang.Object.wait(Object.java:502)
>>>          at
>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>          - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>
>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for 
>>> monitor
>>> entry [0x0185e000]
>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>          at 
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>          at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>          at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>
>>>
>>>          at java.lang.System.checkIO(System.java:253)
>>>          at java.lang.System.setErr(System.java:199)
>>>          at 
>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>
>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>
>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8 
>>> waiting on
>>> condition
>>>
>>> JNI global references: 19
>>>
>>>
>>> Found one Java-level deadlock:
>>> =============================
>>> "Thread-1":
>>>    waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>    which is held by "main"
>>> "main":
>>>    waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>> java.lang.Object),
>>>    which is held by "Thread-1"
>>>
>>> Java stack information for the threads listed above:
>>> ===================================================
>>> "Thread-1":
>>>          at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>          at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>
>>>
>>>          at
>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>          at
>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137) 
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070) 
>>>
>>>          at
>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) 
>>>
>>>
>>>          at 
>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>          at 
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>          at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167) 
>>>
>>>
>>>          at
>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166) 
>>>
>>>
>>>          - locked <0x03ef8b30> (a
>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>          at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>          at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>          at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j 
>>>
>>>
>>> ava:180)
>>>          at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294 
>>>
>>>
>>> )
>>>          at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
>>>
>>>
>>>          at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
>>>
>>>
>>>          at java.lang.Thread.run(Thread.java:744)
>>> "main":
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>          at 
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>          at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>
>>>          at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>
>>>
>>>          at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>          at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>
>>>
>>>          at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>
>>>
>>>          at java.lang.System.checkIO(System.java:253)
>>>          at java.lang.System.setErr(System.java:199)
>>>          at 
>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>
>>> Found 1 deadlock.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20160206/2310a407/attachment-0001.html>


More information about the Concurrency-interest mailing list