output order on parallel collections

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

output order on parallel collections

Russ P.
According to

http://docs.scala-lang.org/overviews/parallel-collections/overview.html

The output order for operations on parallel collections should correspond to the input order:

"The “out of order” semantics of parallel collections only means that the operation will be executed out of order (in a temporal sense. That is, non-sequentially), it does not mean that the result will be re-“combined” out of order (in a spatial sense). On the contrary, results will generally always be reassembled in order– that is, a parallel collection broken into partitions A, B, C, in that order, will be reassembled once again in the order A, B, C. Not some other arbitrary order like B, C, A."

However, I am not finding that to be true. For example, the Vector that results from

    val encounters = for (Trial(trial, note) <- trials) yield
      closestEncounter(trial, flights).copy(note=note)

is not in the same order as the result of

    val encounters = for (Trial(trial, note) <- trials.par) yield
      closestEncounter(trial, flights).copy(note=note)

Am I missing something? Thanks.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: output order on parallel collections

Rex Kerr-2
Order should be preserved.  You can see that this is normally the case with a simple test like

val v = Vector.tabulate(1024*1024)(i => i)
val v2 = v.par.map(i => i*2).toVector
(v2, v2.tail).zipped.forall(_ + 2 == _)

If you have a case where this is _not_ true, and you are not performing any side effects during the computation that could change the identity of what you're copying, then it sounds like a bug.  (I doubt there is a bug; it's probably something else.  But it could be a bug.)

Can you find a minimization?

  --Rex

On Thu, Dec 8, 2016 at 7:06 PM, Russ P. <[hidden email]> wrote:
According to

http://docs.scala-lang.org/overviews/parallel-collections/overview.html

The output order for operations on parallel collections should correspond to the input order:

"The “out of order” semantics of parallel collections only means that the operation will be executed out of order (in a temporal sense. That is, non-sequentially), it does not mean that the result will be re-“combined” out of order (in a spatial sense). On the contrary, results will generally always be reassembled in order– that is, a parallel collection broken into partitions A, B, C, in that order, will be reassembled once again in the order A, B, C. Not some other arbitrary order like B, C, A."

However, I am not finding that to be true. For example, the Vector that results from

    val encounters = for (Trial(trial, note) <- trials) yield
      closestEncounter(trial, flights).copy(note=note)

is not in the same order as the result of

    val encounters = for (Trial(trial, note) <- trials.par) yield
      closestEncounter(trial, flights).copy(note=note)

Am I missing something? Thanks.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: output order on parallel collections

Jason Zaugg

Perhaps you are experiencing SI-9126, which was fixed in Scala 2.11.6.

-jason


On Fri, 9 Dec 2016 at 13:37 Rex Kerr <[hidden email]> wrote:
Order should be preserved.  You can see that this is normally the case with a simple test like

val v = Vector.tabulate(1024*1024)(i => i)
val v2 = v.par.map(i => i*2).toVector
(v2, v2.tail).zipped.forall(_ + 2 == _)

If you have a case where this is _not_ true, and you are not performing any side effects during the computation that could change the identity of what you're copying, then it sounds like a bug.  (I doubt there is a bug; it's probably something else.  But it could be a bug.)

Can you find a minimization?

  --Rex

On Thu, Dec 8, 2016 at 7:06 PM, Russ P. <[hidden email]> wrote:
According to

http://docs.scala-lang.org/overviews/parallel-collections/overview.html

The output order for operations on parallel collections should correspond to the input order:

"The “out of order” semantics of parallel collections only means that the operation will be executed out of order (in a temporal sense. That is, non-sequentially), it does not mean that the result will be re-“combined” out of order (in a spatial sense). On the contrary, results will generally always be reassembled in order– that is, a parallel collection broken into partitions A, B, C, in that order, will be reassembled once again in the order A, B, C. Not some other arbitrary order like B, C, A."

However, I am not finding that to be true. For example, the Vector that results from

    val encounters = for (Trial(trial, note) <- trials) yield
      closestEncounter(trial, flights).copy(note=note)

is not in the same order as the result of

    val encounters = for (Trial(trial, note) <- trials.par) yield
      closestEncounter(trial, flights).copy(note=note)

Am I missing something? Thanks.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: output order on parallel collections

Russ P.
In reply to this post by Rex Kerr-2
I think I figured out what the problem is. If I convert the parallel Vector result back to a regular Vector, then it appears to be in the proper order. Who would have guessed?

--Russ

On Thursday, December 8, 2016 at 7:37:47 PM UTC-8, Rex Kerr wrote:
Order should be preserved.  You can see that this is normally the case with a simple test like

val v = Vector.tabulate(1024*1024)(i => i)
val v2 = v.par.map(i => i*2).toVector
(v2, v2.tail).zipped.forall(_ + 2 == _)

If you have a case where this is _not_ true, and you are not performing any side effects during the computation that could change the identity of what you're copying, then it sounds like a bug.  (I doubt there is a bug; it's probably something else.  But it could be a bug.)

Can you find a minimization?

  --Rex

On Thu, Dec 8, 2016 at 7:06 PM, Russ P. <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="qYJKt5H_BQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">russ.p...@...> wrote:
According to

<a href="http://docs.scala-lang.org/overviews/parallel-collections/overview.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdocs.scala-lang.org%2Foverviews%2Fparallel-collections%2Foverview.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH0W33J_zwi7CNBFqZcUSxd3pRTXw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdocs.scala-lang.org%2Foverviews%2Fparallel-collections%2Foverview.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH0W33J_zwi7CNBFqZcUSxd3pRTXw&#39;;return true;">http://docs.scala-lang.org/overviews/parallel-collections/overview.html

The output order for operations on parallel collections should correspond to the input order:

"The “out of order” semantics of parallel collections only means that the operation will be executed out of order (in a temporal sense. That is, non-sequentially), it does not mean that the result will be re-“combined” out of order (in a spatial sense). On the contrary, results will generally always be reassembled in order– that is, a parallel collection broken into partitions A, B, C, in that order, will be reassembled once again in the order A, B, C. Not some other arbitrary order like B, C, A."

However, I am not finding that to be true. For example, the Vector that results from

    val encounters = for (Trial(trial, note) <- trials) yield
      closestEncounter(trial, flights).copy(note=note)

is not in the same order as the result of

    val encounters = for (Trial(trial, note) <- trials.par) yield
      closestEncounter(trial, flights).copy(note=note)

Am I missing something? Thanks.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="qYJKt5H_BQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">scala-user+...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.