[concurrency-interest] Composing multiple tasks into a single Future/FutureTask

Chris Vest mr.chrisvest at gmail.com
Tue Apr 17 12:21:30 EDT 2012


Hi Ariel,

One thing that came to mind as I read your mail is this:
After the tasks have made their writes, they could create a
CountDownLatch(1), put it on a special stack, and then await it.
Then your fsync'ing thread could operate like this in a loop:
 * pop the entire stack of latches
 * do the fsync
 * count down all the popped latches because their writes are now durable
 * (possibly sleep a little? I don't know...)

This way, the write operations happen before the latch is pushed to the
"sync stack", which happens before the latch is popped, which in turn
happens before the fsync, and then the counting down. Then the task, which
is awaiting on the latch, doesn't complete until the write has been
fsync'ed. This should ensure that no writes are considered durable before
they complete.

I'm not that familiar with guava, so I don't know if it can help you avoid
writing some or all of this infrastructure yourself.

Cheers,
Chris

On Tue, Apr 17, 2012 at 17:39, Ariel Weisberg <ariel at weisberg.ws> wrote:

> Hi all,
>
> I have been looking for the right way to build an API that returns a
> Future that is composed of tasks that will be performed in multiple
> stages, potentially from more than one thread.
>
> My use cases is a disk based persistence store. The initial put API
> call creates a task that compresses and checksums the payload. The
> resulting Future is bound in a Runnable and submitted to a single
> thread ExecutorService that timestamps the entry and appends it to a
> file and it is the Future of the second task that is currently returned
> by the API.
>
>  The trick is that the Future that is returned by the put call should
>  not complete until the write is durable. Writes are made durable by
>  periodically invoking fsync from a dedicated thread or by rotating
>  write threads as each blocks on fsync/force. This allows for group
>  commit at an interval the IO device can handle without killing
>  throughput.
>
> Ideally I could create a FutureTask that is composed of several other
> Futures, some of which will not exist when the FutureTask is initially
> created and returned by the API. I do know how many dependencies there
> will be
> expected up front.
>
> If I had such a thing I would create a task to do the fsync and add it
> as the last dependency to every Future read by the put API call.
>
> This would be a lot easier if I were willing to have many threads
> standing around blocking on Futures, but that rubs me the wrong way. It
> isn't fun to jstack and see all the noise.
>
> I suspect that there is something in Guava
> com.google.common.util.concurrent.Futures to handle this case, but it
> isn't jumping out at me.
>
> Thanks,
> Ariel Weisberg
> _______________________________________________
> Concurrency-interest mailing list
> Concurrency-interest at cs.oswego.edu
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20120417/8e1f6699/attachment.html>


More information about the Concurrency-interest mailing list