[concurrency-interest] Conja-- an accidental jsr166y alternative
dev at davidsoergel.com
Thu Nov 19 02:46:40 EST 2009
I'm sorry I don't entirely understand your questions... in the first case, given that the tasks need to run one at a time for memory reasons, what are you trying to accomplish by running them in a separate thread?
I think your second question asks whether you can exert fine-grained control over the task scheduling, i.e. by explicitly giving different tasks different priorities. Conja doesn't provide for that. It assumes that the tasks are independent of one another (that's what I mean by "functional style"), so there should be no need to impose scheduling constraints among them. In practice, Conja executes the tasks in a given batch in FIFO order. The trick is that nested tasks essentially inherit their priority; this helps tasks finish faster and thus get out of the way in terms of memory. If there is one task somewhere in the call tree that requires a lot of memory, Conja has no way of knowing that in advance, so it will continue to run other tasks on the rest of its worker threads concurrently. It sounds like you'd like to be able to a) postpone a task when you can predict that there's not enough memory currently available to finish it, and b) tell the executor to pause (i.e., finish running tasks but not start any new ones) in hopes of freeing up some memory so that the postponed task can run. These are interesting ideas... am I getting the gist of your problem?
On Nov 18, 2009, at 10:38 PM, Paulo Levi wrote:
> You have cool ideas.
> I "made" (copied most of the concepts) these (terrible) classes for
> scheduling/composing Callables. One of the situations i've found is exactly
> the one you describe in the profileration of executors example you give,
> however i needed to have the requirement that the subtask was only run once
> at a time (huuuuuge memory sink). So i composed my composable callables
> (terrible outdated name it has now) and sent it to a 1 thread executor just
> for that subtask. How would your library handle things like that?
> How about scheduling problems? (i'm having that too - one of my task runs
> the subtask with the huuuuuugee memory requirement, in a LIFO order (with a
> executor from Threads there, however another task - that doesn't know
> anything about the first - also runs a slightly different version of the
> same task/function -> things explode).
> I don't see how could you keep the different Priorities consistent in the
> same Executor (all those task + some that i might execute later should run
> LIFO or FIFO and another should run as soon as possible) or other out there
> On Thu, Nov 19, 2009 at 6:19 AM, David Soergel <dev at davidsoergel.com> wrote:
>> Hi David,
>> I'm not familiar with F# specifically, but you're right that using a
>> functional programming style is the idea here :)
>> Regarding the DepthFirstThreadPoolExecutor, my expectation is that it's
>> only used internally. I didn't just make it private since I figured it
>> might come in handy in some other context, but in fact I've only ever used
>> it from within the Parallel class. Still, I should synchronize it properly
>> just in case. Thanks for catching that!
>> On Nov 18, 2009, at 10:04 PM, David Holmes wrote:
>>> Looks very F# like :)
>>> One thing (and I was just browsing) your DepthFirstThreadPoolExecutor
>>> lifecycle management is not thread-safe. You can leak instances; there's
>>> sync or volatile to ensure visibility of the newly created instance, or
>>> updated cpu counts. There's no sync between shutting down and recreating.
>>> the expectation that this will not be used directly but only from other
>>> library internals which ensures a single-thread manages the executor? If
>>> then you need synchronization in various places.
>>> David Holmes
>>>> -----Original Message-----
>>>> From: concurrency-interest-bounces at cs.oswego.edu
>>>> [mailto:concurrency-interest-bounces at cs.oswego.edu]On Behalf Of David
>>>> Sent: Thursday, 19 November 2009 3:40 PM
>>>> To: concurrency-interest at cs.oswego.edu
>>>> Subject: [concurrency-interest] Conja-- an accidental jsr166y
>>>> Hi all,
>>>> I wrote a Java concurrency library a while back before I knew
>>>> about jsr166y, and finally got around to writing up how it works.
>>>> I realize I'm very late to the party here, but still I hope some
>>>> of the ideas I implemented may interest you.
>>>> The main advantage of my library that I see at the moment is that
>>>> it hides all the hard stuff behind syntactic sugar, and so should
>>>> be very easy for a novice to adopt. I suspect it would be
>>>> straightforward to provide a similarly easy-to-use wrapper around
>>>> the jsr166y internals. I haven't yet done a detailed comparison
>>>> though, so it may well be that jsr166y provides functionality
>>>> that Conja lacks.
>>>> The project home page is at
>>>> http://dev.davidsoergel.com/trac/conja/, and the most important
>>>> design issues are described briefly at
>>>> A couple of those issues are: 1) I schedule nested tasks in
>>>> depth-first order, with advantages much like work stealing; 2) I
>>>> employ various strategies to conserve memory (primarily by not
>>>> leaving tasks waiting around in queues); and 3) I construct
>>>> Runnables lazily and concurrently from an iterator of task
>>>> inputs. One consequence is that "pipelines" consisting of nested
>>>> mapping iterators (i.e., iterators that apply a function to
>>>> elements from an underlying iterator) can be used to provide the
>>>> inputs, in which case the mappings are computed lazily and concurrently.
>>>> I've been using this for some time with excellent performance, so
>>>> I think it works at least :)
>>>> Looking forward to any feedback you may have,
>>>> David Soergel (650) 303-5324
>>>> dev at davidsoergel.com http://www.davidsoergel.com
>>>> Concurrency-interest mailing list
>>>> Concurrency-interest at cs.oswego.edu
>> David Soergel (650) 303-5324
>> dev at davidsoergel.com http://www.davidsoergel.com
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu
David Soergel (650) 303-5324
dev at davidsoergel.com http://www.davidsoergel.com
More information about the Concurrency-interest