[concurrency-interest] implementing a DB write-behind algorithm

studdugie studdugie at gmail.com
Mon May 1 19:39:32 EDT 2006


Another option is to use a CLQ w/ an AtomicBoolean or (now that I
think about it) an AtomicInteger (AI).

The logic goes like this. Thread has database write so it increments
AtomicInteger and compares it against guard variable
(MAX_CONCURR_WRITERS). If AI <= MAX_CONCURR_WRITERS then enter write
loop and drain the CLQ.

The code may look something like this.

void dbPut(DbData data)
{
    dbWriteCLQ.offer( data );

    try
    {
        if(AI.incrementAndGet() <= MAX_CONCURR_WRITERS )
            while(null !=(data = dbWriteCLQ.poll()))
                doDatabaseWrite( data );
    }
    finally
    {
        AI.decrementAndGet();
    }
}

I call it the hijack approach because instead of having one or more
specialist threads to do the job you use a CAS to "hijack" any thread
that happens to be passing by at the wrong (or right) place at the
wrong (or right) time.

Like I said in the first paragraph, you could also use an
AtomicBoolean to do the hijacking (CAS) but that limits you to one
thread.

Obviously hijacking isn't the best solution all the time because
you've pulled a thread away from its normal flow and that flow may be
time sensitive (ex. responding to an HTTP request). But if you have
complete control over every thread that can get hijacked you may be
able to get away w/ it. Your best bet (as always) is slap a profiler
on the code w/ varying loads after you've determined your throughput
minimum.

Love, peace, and hair grease,

Dane

On 4/30/06, Alexandru Popescu <the.mindstorm.mailinglist at gmail.com> wrote:
> Hi!
>
> I firstly have to confess that when getting to concurrency related
> problems, I am getting confused quite quickly :-).
>
> Now, the current problem I am trying to solve is: I am trying to
> figure out how to implement a DB write-behind strategy. Multiple
> processes will post records to be written to the DB, but the actual
> writes should happen on a separate process. So, far I was thinking
> about 2 possible approaches:
> a) continous write-behind: multiple processes write to a queue which
> is continously polled by a separate process. When an element is found
> on the queue, than the write process removes it from queue and
> attempts to write it to the DB.
>
> To have this done, I was looking in the direction of ConcurrentLinkedQueue.
>
> b) batched write-behind: multiple processes post to a size-bounded
> queue. When the max size is reached, the original queue is passed to
> the parallel write process and replaced with a new queue.
>
> To have this done, I was looking in the direction of
> LinkedBlockingQueue with an additional atomic operation of swapping
> the old queue with the new empty one.
>
> My question is: am I looking in the right direction or I am completely
> wrong. Any ideas and help are highly appreciated.
>
> ./alex
> --
> .w( the_mindstorm )p.
>
> _______________________________________________
> Concurrency-interest mailing list
> Concurrency-interest at altair.cs.oswego.edu
> http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
>



More information about the Concurrency-interest mailing list