[concurrency-interest] More control on 'full' with blockingqueue.

Bill Scherer scherer at cs.rochester.edu
Sun Aug 7 12:20:41 EDT 2005


On Sun, 7 Aug 2005, Peter Veentjer - Anchor Men wrote:

> For a searchengine platform I`m currently developing (based on Lucene) I 
> need better control on BlockingQueues when they are full. If I only 
> check on the number of items in the queue it could happen that I would 
> have 10 documents of 100 mb in a Queue (and this is a bad thing). But if 
> I restrict the size of the Queue too much, processing 100 documents of 
> 1kb would take too long. Therefore I need better control if it is 
> allowed to add an item to the Queue and that is why I have written the 
> ConditionalBlockingQueue and it uses a strategy: IsFull (the names have 
> to be improved) and this allows better control. The behaviour of 
> BlockingQueues that block on the capacity (the number of items in the 
> Queue) could be made with the IsFull strategy.
>
> My question is what you think of this solution and if there are others 
> that had the same problem.

I've not had the same problem, but if I'm understanding you correctly, it 
seems that a nice solution might be something along the lines of the 
following:

Modify the various offer() and take() (and related) calls to accept a 
weight parameter (which would also need to be added into the Node class 
(see the insert() method). In your case, you'd supply this as a function 
of the document length. Then, instead of using getAndIncrement() and 
getAndDecrement() calls inside offer() and take(), use getAndAdd() (which 
can take a negative operand) to add the weight of the current document.

With all this done, you have a straightforward way to bound the total size 
of documents being processed at any one time. In particular, your queue 
size plus the size of the largest document you ever process is your 
worst-case size. (It can go over if you remove one document, getting just 
under the queue size, then add something really big. You could also addd 
conditional logic to forbid adding big doucments until the queue is small 
enough to hold them, but I doubt that this would gain you anything in 
practice.

If you need help getting this blocking queue variant working, let me know.

Cheers,
 			-- Bill


More information about the Concurrency-interest mailing list