[concurrency-interest] Conja-- an accidental jsr166y alternative

Paulo Levi i30817 at gmail.com
Thu Nov 19 03:18:14 EST 2009


It's not all of the tasks. As i can compose Callables i offload getting a
image to some services implemented as task. If one fails it tries the other
right?
So i composed the code that should be 1 threaded (including that one huge
one). If that fails i want to run another alternative, but that alternative
is not memory intensive so i can run many threads of that (concurrently
while the first awaits right).
Basically a way to say that tasks of a certain set of type should only
should run on a nº of threads concurrently (a number probably could be
derived from max memory and expected maximum memory use - that is, if you
want to go that way). Your idea of pausing the executor until the others
complete could have the same effect maybe if we just annotated the expected
memory consumption of that task.

For the second case not to complicate too much (its actually used in
conjunction with the first) i'm interested in 2 main things
1) As your library appears to use a singleton executor hidden by a static
method, was there any support at all for different queuing strategies (LIFO
i use), inclusevely across different calls. For example i have a problem in
my app that one of my Executors is FIFO (that contains the huge memory
task), but another functionality (in the main UI thread) calls a Executor
(single-threaded) that calls that task too. What is stumping me in just
having a shared executor like you is - well that new singleThreaded task
should execute as fast as possible (LIFO), but that conflicts with the FIFO
queue i already am using.

On Thu, Nov 19, 2009 at 7:46 AM, David Soergel <dev at davidsoergel.com> wrote:

> Hi Paulo,
>
> I'm sorry I don't entirely understand your questions... in the first case,
> given that the tasks need to run one at a time for memory reasons, what are
> you trying to accomplish by running them in a separate thread?
>
> I think your second question asks whether you can exert fine-grained
> control over the task scheduling, i.e. by explicitly giving different tasks
> different priorities.  Conja doesn't provide for that.  It assumes that the
> tasks are independent of one another (that's what I mean by "functional
> style"), so there should be no need to impose scheduling constraints among
> them.  In practice, Conja executes the tasks in a given batch in FIFO order.
>  The trick is that nested tasks essentially inherit their priority; this
> helps tasks finish faster and thus get out of the way in terms of memory.
>  If there is one task somewhere in the call tree that requires a lot of
> memory, Conja has no way of knowing that in advance, so it will continue to
> run other tasks on the rest of its worker threads concurrently.  It sounds
> like you'd like to be able to a) postpone a task when you can predict that
> there's not enough memory currently available to finish it, and b) tell the
> executor to pause (i.e., finish running tasks but not start any new ones) in
> hopes of freeing up some memory so that the postponed task can run.  These
> are interesting ideas... am I getting the gist of your problem?
>
> -ds
>
>
>
> On Nov 18, 2009, at 10:38 PM, Paulo Levi wrote:
>
> > You have cool ideas.
> > I "made" (copied most of the concepts) these (terrible) classes for
> > scheduling/composing Callables. One of the situations i've found is
> exactly
> > the one you describe in the profileration of executors example you give,
> > however i needed to have the requirement that the subtask was only run
> once
> > at a time (huuuuuge memory sink). So i composed my composable callables
> > (terrible outdated name it has now) and sent it to a 1 thread executor
> just
> > for that subtask. How would your library handle things like that?
> >
> > How about scheduling problems? (i'm having that too - one of my task runs
> > the subtask with the huuuuuugee memory requirement, in a LIFO order (with
> a
> > executor from Threads there, however another task - that doesn't know
> > anything about the first - also runs a slightly different version of the
> > same task/function -> things explode).
> > I don't see how could you keep the different Priorities consistent in the
> > same Executor (all those task + some that i might execute later should
> run
> > LIFO or FIFO and another should run as soon as possible) or other out
> there
> > combinations.
> >
> >
> http://code.google.com/p/bookjar-utils/source/browse/BookJar-utils/src/util/ChainRunnable.java
> >
> http://code.google.com/p/bookjar-utils/source/browse/BookJar-utils/src/util/Threads.java
> >
> > On Thu, Nov 19, 2009 at 6:19 AM, David Soergel <dev at davidsoergel.com>
> wrote:
> >
> >> Hi David,
> >>
> >> I'm not familiar with F# specifically, but you're right that using a
> >> functional programming style is the idea here :)
> >>
> >> Regarding the DepthFirstThreadPoolExecutor, my expectation is that it's
> >> only used internally.  I didn't just make it private since I figured it
> >> might come in handy in some other context, but in fact I've only ever
> used
> >> it from within the Parallel class.  Still, I should synchronize it
> properly
> >> just in case. Thanks for catching that!
> >>
> >> -ds
> >>
> >>
> >> On Nov 18, 2009, at 10:04 PM, David Holmes wrote:
> >>
> >>> David,
> >>>
> >>> Looks very F# like :)
> >>>
> >>> One thing (and I was just browsing) your DepthFirstThreadPoolExecutor
> >>> lifecycle management is not thread-safe. You can leak instances;
> there's
> >> no
> >>> sync or volatile to ensure visibility of the newly created instance, or
> >> of
> >>> updated cpu counts. There's no sync between shutting down and
> recreating.
> >> Is
> >>> the expectation that this will not be used directly but only from other
> >>> library internals which ensures a single-thread manages the executor?
> If
> >> not
> >>> then you need synchronization in various places.
> >>>
> >>> Cheers,
> >>> David Holmes
> >>>
> >>>> -----Original Message-----
> >>>> From: concurrency-interest-bounces at cs.oswego.edu
> >>>> [mailto:concurrency-interest-bounces at cs.oswego.edu]On Behalf Of David
> >>>> Soergel
> >>>> Sent: Thursday, 19 November 2009 3:40 PM
> >>>> To: concurrency-interest at cs.oswego.edu
> >>>> Subject: [concurrency-interest] Conja-- an accidental jsr166y
> >>>> alternative
> >>>>
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I wrote a Java concurrency library a while back before I knew
> >>>> about jsr166y, and finally got around to writing up how it works.
> >>>> I realize I'm very late to the party here, but still I hope some
> >>>> of the ideas I implemented may interest you.
> >>>>
> >>>> The main advantage of my library that I see at the moment is that
> >>>> it hides all the hard stuff behind syntactic sugar, and so should
> >>>> be very easy for a novice to adopt.  I suspect it would be
> >>>> straightforward to provide a similarly easy-to-use wrapper around
> >>>> the jsr166y internals.  I haven't yet done a detailed comparison
> >>>> though, so it may well be that jsr166y provides functionality
> >>>> that Conja lacks.
> >>>>
> >>>> The project home page is at
> >>>> http://dev.davidsoergel.com/trac/conja/, and the most important
> >>>> design issues are described briefly at
> >>>> http://dev.davidsoergel.com/trac/conja/wiki/PrinciplesOfOperation
> >>>>
> >>>> A couple of those issues are: 1) I schedule nested tasks in
> >>>> depth-first order, with advantages much like work stealing; 2) I
> >>>> employ various strategies to conserve memory (primarily by not
> >>>> leaving tasks waiting around in queues); and 3) I construct
> >>>> Runnables lazily and concurrently from an iterator of task
> >>>> inputs.  One consequence is that "pipelines" consisting of nested
> >>>> mapping iterators (i.e., iterators that apply a function to
> >>>> elements from an underlying iterator) can be used to provide the
> >>>> inputs, in which case the mappings are computed lazily and
> concurrently.
> >>>>
> >>>> I've been using this for some time with excellent performance, so
> >>>> I think it works at least :)
> >>>>
> >>>> Looking forward to any feedback you may have,
> >>>>
> >>>> -ds
> >>>>
> >>>>
> >>>> _______________________________________________________
> >>>> David Soergel                            (650) 303-5324
> >>>> dev at davidsoergel.com        http://www.davidsoergel.com
> >>>> _______________________________________________________
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Concurrency-interest mailing list
> >>>> Concurrency-interest at cs.oswego.edu
> >>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
> >>>
> >>
> >>
> >> _______________________________________________________
> >> David Soergel                            (650) 303-5324
> >> dev at davidsoergel.com        http://www.davidsoergel.com
> >> _______________________________________________________
> >>
> >>
> >> _______________________________________________
> >> Concurrency-interest mailing list
> >> Concurrency-interest at cs.oswego.edu
> >> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
> >>
>
>
> _______________________________________________________
> David Soergel                            (650) 303-5324
> dev at davidsoergel.com        http://www.davidsoergel.com
> _______________________________________________________
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.oswego.edu/pipermail/concurrency-interest/attachments/20091119/8683fb03/attachment-0001.html>


More information about the Concurrency-interest mailing list