[concurrency-interest] ForkJoinPool not designed for nested Java 8 streams.parallel().forEach( ... )

Christian Fries email at christian-fries.de
Fri May 9 09:05:05 EDT 2014


Dear Alex.

Example: The Levenberg-Marquardt non-linear least square optimization can be parallelized easily on arguments of the objective function x -> f(x). Here f(x) is a function an the LM class performs parallel valuation on a stream of arguments. The function is provided as a lambda or abstract class. In many applications x is a vector and the function f is itself a composite of independent components which can be calculated in parallel.

Hence I have two classes: F (the objective function) and LM (the optimized) and f is provided to LM will introduce nested parallelism.

This is a real application. I already have this working perfectly in Java 6 with nested parallelism (without FJP).

Wouldn’t flattening mean loss of abstraction (no abstraction between F and LM), spaghetti code, etc.

Christian

PS: I am really surprised that there is so much discussion on this list in that direction. Modern frameworks are full of nested parallelism. Did anybody check the code that I have posted and confirm the behavior?

Am 09.05.2014 um 14:46 schrieb Oleksandr Otenko <oleksandr.otenko at oracle.com>:

> To be honest, I don't understand why "nested" parallellism occurs at all.
> 
> 1. flatten
> 2. recreate the shape
> 
> unless you are dealing with infinite streams; in the latter case the order of processing is extremely important, and it is no longer "uncontrolled" parallellism.
> 
> Alex
> 
> On 08/05/2014 09:48, Christian Fries wrote:
>> Dear All.
>> 
>> With respect to the improvement of the documentation suggested
>> 
>> Am 06.05.2014 um 01:09 schrieb Doug Lea <dl at cs.oswego.edu>:
>> 
>>> Sorry that you and a bunch of people at stackOverflow spent so much time
>>> on this because you did not did not notice the java.util.stream documentation
>>> (http://docs.oracle.com/javase/8/docs/api/java/util/stream/package-summary.html)
>>> about the use of independent functions/actions (especially not those including
>>> synchronization). We should do something about this.
>> I would like to add two aspects from a user perspective:
>> 
>> First, the linked page ( http://docs.oracle.com/javase/8/docs/api/java/util/stream/package-summary.html ) does not mention the ForkJoinPool or Managed Blocker, etc. at all. The words „pool“, „common“, „fork“, „managed“ do not even occur in the document. It is just a discussion of parallel streams from the stream perspective suggesting that code like the one I have used (thread save, no interference, etc.) works fine. (This is not actually my observation, it was a comment on SO).
>> So a footnote that the backend of parallel streams is currently the FJP and that corresponding limitations apply (use of managed blockers) should be added (on the other hand - I would prefer to have the streams framework work without too much knowledge of the underlying implementation).
>> 
>> Second, I would like to comment on the idea to „forbid“ nested parallelism at all: From a user perspective I try to (or have to) use parallelism a lot. In my library many parts are functional, stateless, free of interference. If you use parallelism a lot it may be that you have parallelism encapsulated „inside“ many small parts and nested parallelism is a natural thing. It is also a good thing. If I have an outer parallel loop of 5 on our 64 core machine it would be a waste to forbid inner (nested) parallelism. Everyone of the 5 could work on its task with a parallelism of > 10 - so we are talking about a factor of 10. - I have this situation in my lib and it works great.
>> 
>> Of course, one aspect of the issue in my previous post is that we are submitting to a COMMON thread pool. A workaround would be to use a dedicated pool for each inner loop. But a common thread pool is an advantage, since it allows to globally control parallelism!
>> 
>> Best
>> Christian
>> 
>> 
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
> 




More information about the Concurrency-interest mailing list