[concurrency-interest] ForkJoinPool not designed for nested Java 8 streams.parallel().forEach( ... )
oleksandr.otenko at oracle.com
Fri May 9 10:51:29 EDT 2014
It never hurts to see if your abstraction meets the expectation of what
FJP can do.
You have a tree of expressions that must be beta-reduced. The meaning of
"nested" is undefined, and the order of evaluation is meaningless for
"eager" evaluation strategies.
"Flattening" here means representing the tree as a list of expressions
that need reducing. "Reconstructing" here means placing the reduction
result back into the same structure.
If you don't expect FJP to work out arbitrary graph of expressions that
needs reducing, then you need to construct the graph and reduce it yourself.
On 09/05/2014 14:05, Christian Fries wrote:
> Dear Alex.
> Example: The Levenberg-Marquardt non-linear least square optimization can be parallelized easily on arguments of the objective function x -> f(x). Here f(x) is a function an the LM class performs parallel valuation on a stream of arguments. The function is provided as a lambda or abstract class. In many applications x is a vector and the function f is itself a composite of independent components which can be calculated in parallel.
> Hence I have two classes: F (the objective function) and LM (the optimized) and f is provided to LM will introduce nested parallelism.
> This is a real application. I already have this working perfectly in Java 6 with nested parallelism (without FJP).
> Wouldn’t flattening mean loss of abstraction (no abstraction between F and LM), spaghetti code, etc.
> PS: I am really surprised that there is so much discussion on this list in that direction. Modern frameworks are full of nested parallelism. Did anybody check the code that I have posted and confirm the behavior?
> Am 09.05.2014 um 14:46 schrieb Oleksandr Otenko <oleksandr.otenko at oracle.com>:
>> To be honest, I don't understand why "nested" parallellism occurs at all.
>> 1. flatten
>> 2. recreate the shape
>> unless you are dealing with infinite streams; in the latter case the order of processing is extremely important, and it is no longer "uncontrolled" parallellism.
>> On 08/05/2014 09:48, Christian Fries wrote:
>>> Dear All.
>>> With respect to the improvement of the documentation suggested
>>> Am 06.05.2014 um 01:09 schrieb Doug Lea <dl at cs.oswego.edu>:
>>>> Sorry that you and a bunch of people at stackOverflow spent so much time
>>>> on this because you did not did not notice the java.util.stream documentation
>>>> about the use of independent functions/actions (especially not those including
>>>> synchronization). We should do something about this.
>>> I would like to add two aspects from a user perspective:
>>> First, the linked page ( http://docs.oracle.com/javase/8/docs/api/java/util/stream/package-summary.html ) does not mention the ForkJoinPool or Managed Blocker, etc. at all. The words „pool“, „common“, „fork“, „managed“ do not even occur in the document. It is just a discussion of parallel streams from the stream perspective suggesting that code like the one I have used (thread save, no interference, etc.) works fine. (This is not actually my observation, it was a comment on SO).
>>> So a footnote that the backend of parallel streams is currently the FJP and that corresponding limitations apply (use of managed blockers) should be added (on the other hand - I would prefer to have the streams framework work without too much knowledge of the underlying implementation).
>>> Second, I would like to comment on the idea to „forbid“ nested parallelism at all: From a user perspective I try to (or have to) use parallelism a lot. In my library many parts are functional, stateless, free of interference. If you use parallelism a lot it may be that you have parallelism encapsulated „inside“ many small parts and nested parallelism is a natural thing. It is also a good thing. If I have an outer parallel loop of 5 on our 64 core machine it would be a waste to forbid inner (nested) parallelism. Everyone of the 5 could work on its task with a parallelism of > 10 - so we are talking about a factor of 10. - I have this situation in my lib and it works great.
>>> Of course, one aspect of the issue in my previous post is that we are submitting to a COMMON thread pool. A workaround would be to use a dedicated pool for each inner loop. But a common thread pool is an advantage, since it allows to globally control parallelism!
>>> Concurrency-interest mailing list
>>> Concurrency-interest at cs.oswego.edu
More information about the Concurrency-interest