[concurrency-interest] DirectByteBuffers and reachabilityFence

Alexandre De Champeaux adc at quartetfs.com
Mon Dec 7 12:46:23 EST 2015


Yep things are fine if I add a call to a method similar to the position()
method of DBB after calling copyToArray().

On Mon, Dec 7, 2015 at 6:23 PM, Vitaly Davidovich <vitalyd at gmail.com> wrote:

> Can you more closely mirror DBB by updating some state of your
> SimpleBuffer after calling copyToArray()? DBB returns 'this' from its
> get(byte[], ...) and also sets position after the Bits.copyToArray() method.
>
> On Mon, Dec 7, 2015 at 12:08 PM, Alexandre De Champeaux <adc at quartetfs.com
> > wrote:
>
>> I have actually written a test class that reproduces a segfault. It uses
>> an structure similar to DBB and an updated version of the Bits.copyToArray
>> method which forces a GC and waits a bit from within it to simulate a
>> safepoint stop and a GC, and leaves time to the reference handler to clean
>> up.
>> Is this test too far away from the DBB code to actually simulate a
>> possible behavior of DBB? Or is hotspot only able to not mark as
>> unreachable vanilla java DBB, but not customer classes?
>>
>> Here is the test class:
>>
>>
>> import sun.misc.Cleaner;
>> import sun.misc.Unsafe;
>>
>> /**
>>  * This test checks whether or not the JVM will dereference "this" and
>> destroy the buffer while
>>  * accessing it.
>>  * <p>
>>  * Run the main method of this class to run the test.
>>  * <p>
>>  * Note that the test segfaults each time if using a JNI call to
>> mmap/munmap to do the memory
>>  * management but not as often using Unsafe (malloc recycling memory?).
>>  * <p>
>>  * Example output of a failure:
>>  *
>>  * <pre>
>> TestGcThisAndDestroyBuffer.main() 0 -- 0
>> TestGcThisAndDestroyBuffer.main() 0 -- 1
>> TestGcThisAndDestroyBuffer.main() 0 -- 2
>> TestGcThisAndDestroyBuffer.main() 0 -- 3
>> TestGcThisAndDestroyBuffer.main() 0 -- 4
>> TestGcThisAndDestroyBuffer.main() 0 -- 5
>> TestGcThisAndDestroyBuffer.main() 0 -- 6
>> TestGcThisAndDestroyBuffer.main() 0 -- 7
>> TestGcThisAndDestroyBuffer.main() 0 -- 8
>> TestGcThisAndDestroyBuffer.main() 0 -- 9
>> TestGcThisAndDestroyBuffer.main() 0 -- 10
>> TestGcThisAndDestroyBuffer.main() 0 -- 11
>> TestGcThisAndDestroyBuffer.main() 0 -- 12
>> TestGcThisAndDestroyBuffer.main() 0 -- 13
>> TestGcThisAndDestroyBuffer.main() 0 -- 14
>> Must be compiled now, start doing some GCs 30001
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> #  SIGSEGV (0xb) at pc=0x00007f489fb86220, pid=16949, tid=139949914818304
>> #
>> # JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build
>> 1.8.0_60-b27)
>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode
>> linux-amd64 compressed oops)
>> # Problematic frame:
>> # V  [libjvm.so+0x7f7220]
>> #
>> # Failed to write core dump. Core dumps have been disabled. To enable
>> core dumping, try "ulimit -c unlimited" before starting Java again
>> #
>> # An error report file with more information is saved as:
>> #
>> /SSDhome/alexandre/ActivePivot5/scripts/adc/java_tests/hs_err_pid16949.log
>> Compiled method (nm)   10180   18     n 0
>> sun.misc.Unsafe::copyMemory (native)
>> total in heap  [0x00007f488910f2d0,0x00007f488910f640] = 880
>> relocation     [0x00007f488910f3f8,0x00007f488910f440] = 72
>> main code      [0x00007f488910f440,0x00007f488910f640] = 512
>> Compiled method (nm)   10180   18     n 0
>> sun.misc.Unsafe::copyMemory (native)
>> total in heap  [0x00007f488910f2d0,0x00007f488910f640] = 880
>> relocation     [0x00007f488910f3f8,0x00007f488910f440] = 72
>> main code      [0x00007f488910f440,0x00007f488910f640] = 512
>> #
>> # If you would like to submit a bug report, please visit:
>> #   http://bugreport.java.com/bugreport/crash.jsp
>> #
>>  * </pre>
>>  */
>> @SuppressWarnings("restriction")
>> public class TestGcThisAndDestroyBuffer {
>>
>> /** The threshold found in Bits.copyMemory */
>> static final long UNSAFE_COPY_THRESHOLD = 1024L * 1024L;
>> private static final Unsafe unsafe = retrieveUnsafe();
>>
>> static long ctr = 0;
>>
>> // The tested method
>> // See the method in Bits.copyToArray
>> /* For Bits.copyToArray to segfault we need:
>> *
>> * 1) The buffer to be garbage collected while calling the method, for
>> that we need:
>> * - A safepoint exist in the method
>> * - The buffer needs to not be held, so JIT compil must have occured,
>> * and JIT must have dereferenced the DBB at the time of the call
>> * 2) The reference handler thread to have run the associated cleaner
>> */
>> static void copyToArray(long srcAddr, Object dst, long dstBaseOffset,
>> long dstPos, long length) {
>> long offset = dstBaseOffset + dstPos;
>> while (length > 0) {
>> // Code differing from Bits.
>> if (ctr++ >= 30_000) {
>> // Counter is there to wait for JIT compilation
>> System.out.println("Must be compiled now, start doing some GCs " + ctr);
>> // Here we simulate a safepoint, and a GC at this safepoint
>> System.gc();
>> try {
>> // And here we wait for the reference handler to perform the cleaning to
>> simulate a fast cleaning.
>> Thread.sleep(1000);
>> } catch (InterruptedException e) {
>> Thread.currentThread().interrupt();
>> }
>> }
>>
>> long size = (length > UNSAFE_COPY_THRESHOLD) ? UNSAFE_COPY_THRESHOLD :
>> length;
>> unsafe.copyMemory(null, srcAddr, dst, offset, size);
>> length -= size;
>> srcAddr += size;
>> offset += size;
>> }
>> }
>>
>> // A basic class inspired from java.nio.DirectByteBuffer
>> static class SimpleBuffer {
>> static int SIZE = (int) (2 * UNSAFE_COPY_THRESHOLD);
>> static final long arrayBaseOffset = (long)
>> unsafe.arrayBaseOffset(byte[].class);
>>
>> final long addr;
>>
>> public SimpleBuffer() {
>> addr = unsafe.allocateMemory(SIZE);
>> unsafe.setMemory(addr, SIZE, (byte) 0);
>> Cleaner.create(this, new Deallocator(addr));
>> }
>>
>> public byte[] copy() {
>> final byte[] a = new byte[SIZE];
>> copyToArray(addr, a, arrayBaseOffset, 0, SIZE);
>> return a;
>> }
>>
>> private static class Deallocator implements Runnable {
>>
>> private long address;
>>
>> private Deallocator(long address) {
>> assert (address != 0);
>> this.address = address;
>> }
>>
>> @Override
>> public void run() {
>> if (address == 0) {
>> // Paranoia
>> return;
>> }
>> unsafe.freeMemory(address);
>> address = 0;
>> }
>>
>> }
>>
>> }
>>
>> static byte[] aBunchOfCopies() {
>> byte[] res = null;
>> for (int i = 0; i < 1000; ++i) {
>> res = new SimpleBuffer().copy();
>> if (res[0] == -1) {
>> return res;
>> }
>> }
>> return res;
>> }
>>
>> public static void main(String[] args) {
>> int i = 0;
>> while (true) {
>> final byte[] res = aBunchOfCopies();
>> System.out.println("TestGcThisAndDestroyBuffer.main() " + res[0] + " -- "
>> + i++);
>> }
>> }
>>
>> private static final sun.misc.Unsafe retrieveUnsafe() {
>> try {
>> return sun.misc.Unsafe.getUnsafe();
>> } catch (SecurityException se) {
>> try {
>> return java.security.AccessController
>> .doPrivileged(new
>> java.security.PrivilegedExceptionAction<sun.misc.Unsafe>() {
>> @Override
>> public sun.misc.Unsafe run() throws Exception {
>> java.lang.reflect.Field f =
>> sun.misc.Unsafe.class.getDeclaredField("theUnsafe");
>> f.setAccessible(true);
>> return (sun.misc.Unsafe) f.get(null);
>> }
>> });
>> } catch (java.security.PrivilegedActionException e) {
>> throw new RuntimeException("Could not initialize intrinsics",
>> e.getCause());
>> }
>> }
>> }
>>
>> }
>>
>>
>> On Mon, Dec 7, 2015 at 4:15 PM, Vitaly Davidovich <vitalyd at gmail.com>
>> wrote:
>>
>>> JIT knows about Unsafe operations, and it also knows the type of memory
>>> being accessed (or sometimes knows it doesn't know :)).  So I don't think
>>> it'll mark a DBB as unreachable while these operations are in-flight.
>>>
>>> Peter's scenario is unique to WeakReference since it's intentionally not
>>> considered a strong reference and there's otherwise plain java code in his
>>> example (that JIT can reason about easily otherwise).
>>>
>>> sent from my phone
>>> On Dec 7, 2015 8:10 AM, "Alexandre De Champeaux" <adc at quartetfs.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I recently had a look at the discussion started by Peter Levart on
>>>> October 21 (
>>>> http://cs.oswego.edu/pipermail/concurrency-interest/2015-October/014493.html
>>>> ).
>>>>
>>>> It was a very insightful discussion, and made me aware that the "this"
>>>> object could be garbage collected while being inside a call of one of its
>>>> method.
>>>>
>>>> However, this got me concerned about the java.nio.DirectByteBuffer read
>>>> and write methods:
>>>> If the "this" object is garbage collected when making a call like
>>>> ByteBuffer.allocateDirect(int).someGetOrPutMethod(), the native
>>>> pointer that is passed to sun.misc.Unsafe might be freed, and accessing it
>>>> will cause the read/write to occur in an invalid memory area, which might
>>>> lead to a segfault, or other major issues.
>>>> This would be quite unlikely: JIT compilation needs to occur while
>>>> keeping a safepoint, then a GC must happen, and finally the
>>>> ReferenceHandler thread needs to perform its cleanup fast enough.
>>>>
>>>> I am particularly concerned by the get(byte[] dst, int offset, int
>>>> length) method, that in turns calls Bits.copyToArray, which purposely
>>>> splits its calls to Unsafe.copyMemory to allow for safepoints to sneak in.
>>>>
>>>> Am I correct, or does the JVM performs specific protection for
>>>> instances of DirectByteBuffer?
>>>>
>>>> Regards,
>>>>
>>>> Alexandre de Champeaux
>>>>
>>>> _______________________________________________
>>>> Concurrency-interest mailing list
>>>> Concurrency-interest at cs.oswego.edu
>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>
>>>>
>>
>>
>> --
>> Alexandre
>>
>
>


-- 
Alexandre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20151207/1a5ab067/attachment-0001.html>


More information about the Concurrency-interest mailing list