[concurrency-interest] LinkedBlockingQueue extract and memory "waste"

Francois Valdy francois.valdy at gmail.com
Wed Feb 11 05:10:52 EST 2009


Sure here it is:
I know it's a dummy use case, but still a valid one.
LinkedBlockingQueues that survive a full gc will inevitably make old
gen grow, even if empty !
Following test will then output:
Test 1 took: 1595ms
Test 2 took: 2639ms
Exception in thread "main" java.lang.AssertionError: Full GC has
occured during the test: 47 times
	at TestLBQ.main(TestLBQ.java:72)

My workaround of null'ing head.next before moving the head works
perfectly and displays (as expected):
Test 1 took: 1629ms
Test 2 took: 1592ms

I understand it's a hard decision to make to change code based on JVM
implementation but still I'd rather nullify it.

If you want to be really frightened, see the output for G1 (JVM 1.7
b44 with -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC)

JDK7 version:
Test 1 took: 5456ms
Test 2 took: 5575ms

With null assignment:
Test 1 took: 1698ms
Test 2 took: 1602ms

D'oh !

public class TestLBQ
{
    static GarbageCollectorMXBean fullgcx;
    static
    {
        for (GarbageCollectorMXBean gcx :
ManagementFactory.getGarbageCollectorMXBeans().toArray(new
GarbageCollectorMXBean[2]))
        {
            if (gcx == null) continue; // G1 MXBeans aren't defined yet it seems
            if ("PS MarkSweep".equals(gcx.getName()) ||
"MarkSweepCompact".equals(gcx.getName()))
            {
                fullgcx = gcx;
            }
            else if ("ConcurrentMarkSweep".equals(gcx.getName()))
            {
                throw new AssertionError("Test to be executed with PS
MarkSweep (standard old collector)");
            }
        }
    }

    public static void main(String[] args) throws InterruptedException
    {
        long fullgc_count_ref=0, fullgc_count=0, start, end;

        // first test, create the queue after GC, head node will be in young gen
        System.gc();
        LinkedBlockingQueue q = new LinkedBlockingQueue();
        if (fullgcx != null) fullgc_count_ref = fullgcx.getCollectionCount();
        Object DUMMY = new Object();
        start = System.currentTimeMillis();
        for (int i = 0; i < 10000000; i++)
        {
            q.offer(DUMMY);
            q.take();
        }
        System.out.println("Test 1 took: " +
(System.currentTimeMillis()-start) + "ms");
        if (fullgcx != null) fullgc_count = fullgcx.getCollectionCount();
        if (fullgc_count > fullgc_count_ref)
            throw new AssertionError("Full GC has occured during the
test: " + (fullgc_count-fullgc_count_ref) + " times"); // not thrown

        // SAME test, but create the queue before GC, head node will
be in old gen
        q = new LinkedBlockingQueue();
        System.gc();
        if (fullgcx != null) fullgc_count_ref = fullgcx.getCollectionCount();
        start = System.currentTimeMillis();
        for (int i = 0; i < 10000000; i++)
        {
            q.offer(DUMMY);
            q.take();
        }
        System.out.println("Test 2 took: " +
(System.currentTimeMillis()-start) + "ms");
        if (fullgcx != null) fullgc_count = fullgcx.getCollectionCount();
        if (fullgc_count > fullgc_count_ref)
            throw new AssertionError("Full GC has occured during the
test: " + (fullgc_count-fullgc_count_ref) + " times"); // ERROR
    }
}


On Wed, Feb 11, 2009 at 8:56 AM, Martin Buchholz <martinrb at google.com> wrote:
> Hi Francois,
>
> Your idea is very interesting.  I don't recall having seen it before.
> Is this a new rule of thumb, to null out fields of old about-to-become
> garbage objects if they are pointing at newer objects?
>
> Can you produce a synthetic benchmark where nulling out
> the next field doubles the performance?  That might get Doug
> to agree with you.
>
> Martin
>
> On Tue, Feb 10, 2009 at 13:26, Francois Valdy <francois.valdy at gmail.com> wrote:
>> Hi,
>>
>> I'm having memory issues with LinkedBlockingQueue (don't worry, I
>> won't come along saying there's a memory leak in the JDK).
>>
>> The issue is that the method extract() doesn't "help" the GC by
>> dereferencing head.next prior to changing the head node (with
>> head.next = null;).
>>
>> What comes then is that once a node has been put in old gen for
>> whatever reason, all generated nodes from this LinkedBlockingQueue
>> will be as well.
>> You'll tell me that memory will be recovered upon a full gc, but my
>> point is exactly to avoid full gc.
>>
>> Easiest way to reproduce issue is for force a gc (System.gc(), or
>> JConsole/JVisualVM) during queue usage, as it'll promote the current
>> used nodes to old gen.
>> Prior to this step memory was recovered smoothly by minor gc's, after
>> this step you're good for a full gc of only LinkedBlockingQueue nodes
>> (16bytes in JRE32, 32bytes in JRE64).
>>
>> Doug, I understand your attempt at removing extraneous code, but maybe
>> this one wasn't the best choice :)
>> LinkedList for instance doesn't follow the same strategy, and
>> explicitly dereferences next/previous (even if unnecessary from a
>> memory leak pov).
>>
>> I'll now look for a workaround (might end up "recoding"
>> LinkedBlockingQueue with one added line :) )
>>
>> Any thoughts ? (I won't consider moving to realtime java for just that :) )
>>
>> Cheers.
>>
>> (this might be the reason for all those unresolved threads about
>> LinkedBlockingQueue memory consumption, just a guess)
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>
>



More information about the Concurrency-interest mailing list