[concurrency-interest] More control on 'full' with blockingqueue.
scherer at cs.rochester.edu
Sun Aug 7 12:20:41 EDT 2005
On Sun, 7 Aug 2005, Peter Veentjer - Anchor Men wrote:
> For a searchengine platform I`m currently developing (based on Lucene) I
> need better control on BlockingQueues when they are full. If I only
> check on the number of items in the queue it could happen that I would
> have 10 documents of 100 mb in a Queue (and this is a bad thing). But if
> I restrict the size of the Queue too much, processing 100 documents of
> 1kb would take too long. Therefore I need better control if it is
> allowed to add an item to the Queue and that is why I have written the
> ConditionalBlockingQueue and it uses a strategy: IsFull (the names have
> to be improved) and this allows better control. The behaviour of
> BlockingQueues that block on the capacity (the number of items in the
> Queue) could be made with the IsFull strategy.
> My question is what you think of this solution and if there are others
> that had the same problem.
I've not had the same problem, but if I'm understanding you correctly, it
seems that a nice solution might be something along the lines of the
Modify the various offer() and take() (and related) calls to accept a
weight parameter (which would also need to be added into the Node class
(see the insert() method). In your case, you'd supply this as a function
of the document length. Then, instead of using getAndIncrement() and
getAndDecrement() calls inside offer() and take(), use getAndAdd() (which
can take a negative operand) to add the weight of the current document.
With all this done, you have a straightforward way to bound the total size
of documents being processed at any one time. In particular, your queue
size plus the size of the largest document you ever process is your
worst-case size. (It can go over if you remove one document, getting just
under the queue size, then add something really big. You could also addd
conditional logic to forbid adding big doucments until the queue is small
enough to hold them, but I doubt that this would gain you anything in
If you need help getting this blocking queue variant working, let me know.
More information about the Concurrency-interest