Scala Language › Scala - User

Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

8 messages Options

kraythe

Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

Although I cannot use the Scala language at work because the productivity of our team would crater, I have still an interest in a language I would love to be able to use in a business world. Deciding that perhaps I was too harsh on the language, I have been playing with a couple of Scala libraries to try and give the language another shot in my off hours work. I just spent the better part of 7 hours playing with Slick 3.2.0 in Akka and Scala. I was massively frustrated and it turned out that yet again it is an implicit conversion that hosed me. Consider the following slick project.

Users.scala

package persistence.entities

import slick.driver.H2Driver.api._

case class User(id: Long, name: String)

class UserTable(tag: Tag) extends Table[User](tag, "user") {
  def id = column[Long]("id", O.PrimaryKey, O.AutoInc)

  def name = column[String]("name")

  def * = (id, name) <> (User.tupled, User.unapply)
}

UsersDAL.scala

package persistence.dal

import persistence.entities.{Tables, User, UserTable}
import slick.jdbc.JdbcProfile
import slick.lifted.Query

import scala.concurrent.Future

trait UserDAL {
  def findById(id: Long) : Future[Option[User]]
}


class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL {
  override def findById(userId: Long)  = null

  def foo (userId : Long) = {
    val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId)
    db.run(filter.result)
  }
}

Should be simple right? No. The db.run(filter.result) line does not compile on the result call. OK, time to comb the Slick documentation. No dice, my code looks just like theirs does fundamentally. So why is the result method not available. Do I have the wrong type? Nope. What about the wrong library? Nope. Update from 3.1.1 to 3.2.0 ? Done ... but ... no. What about the wrong structure for my table? Am I missing something? Comb example code ... and nope. OK, I am frustrated as all get out. In the meantime my irritation grows when I see example code like the following.

 override def insert(rows: Seq[A]): Future[Seq[Long]] = {
    db.run(tableQ returning tableQ.map(_.id) ++= rows.filter(_.isValid))
  }

Good god, how to decode that mess. Obviously run is taking something as a parameter but exactly what is the order of operations here? Anyway, I digress.

Back to the subject at hand, I am at a complete loss for how to explain the problem here so, I decide to create a post on the Slick forums to figure out what I am doing wrong. I think, for simplicity sake lets combine all of our classes into one file.

package persistence.entities

import java.time.Instant
import slick.driver.H2Driver.api._
import slick.jdbc.JdbcProfile
import slick.lifted.{Query, TableQuery}
import scala.concurrent.Future

case class User(id: Long, name: String)

class UserTable(tag: Tag) extends Table[User](tag, "user") {
  def id = column[Long]("id", O.PrimaryKey, O.AutoInc)

  def name = column[String]("name")

  def * = (id, name) <> (User.tupled, User.unapply)
}

object Users {
  val users = TableQuery[UserTable]

  trait UserDAL {
    def findById(id: Long) : Future[Option[User]]
  }

  class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL {
    override def findById(userId: Long)  = null

    def foo (userId : Long) = {
      val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId)
      db.run(filter.result)
    }
  }
}

OK, now that we have combined them we are ready to post this .. but wait ... it compiles now. What The Fork? Oh wait. There was an implicit conversion going on in there somewhere. There was something in the new file converting our query type into something that had the result method predefined. Magically, behind the scenes and with the right import it doesn't happen and the code doesn't compile.After experimentally deleting imports i discover it's import slick.driver.H2Driver.api._ that is the home of the implicit conversion. So I go look at the file and nope, no implicit. I drill and drill and finally find it. Wonderful. 7+ hours wasted for this. The fun part is the implict conversion happened as a result of the implicit conversion of an implicit conversion of an implicit conversion. Do the designers of Scala think this is actually good? You are just supposed to "know" that the conversion of a conversion of a conversion is happening I guess. Or is the goal to make the code so cryptic and opaque that the source of the library is entirely unreadable?

Given my original UsersDAL file there was no way for me to know that to make the code compile I had to find some implict conversion that could turn a Query into a DBIOAction. There is no means for you to know where this conversion is, what file has to be imported. All of the example code just assumes you must already know. Its fine copying the example code but if you modify, reorganize code into many files you break the delicately balanced Jenga tower and it doesn't work.

So after playing around for a few days with this project, I can see that if I ever brought this code into a business environment my CEO would absolutely murder me as it would take 6 months to add the most basic of features, not to mention 1+ years to train a new developer to be even moderately productive. Once again, its the implicit and its use in the language that is at the core of the problem. It actually makes me sad, but it would be wholesale irresponsible of me as a Principal Software Architect to recommend these techs in a business that makes its money putting out product for users in a short time.

So why post this? Because otherwise i LIKE Scala and wish it could be different. But its becoming like the JDK, they have decided that is the direction they are going and no one on the planet can convince them otherwise. OK now that I am done stating my opinion, I am ready to get flamed again. :-)

-- Robert Simmons Jr.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Patrik Andersson-3

Re: Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

I can really agree with you that Slick’s documentation leaves a little to be desired. The content is all there but hard to navigate.

But you have in fact _not_ followed their way of implementing code. You are directly importing the driver profile but forgetting the api._-import. Sure, you get very hard to decipher error messages - I wonder if something can be done about that. But you are still missing an import.

I’ve spent/ wasted, a matter of perspective really, a lot of time on Slick and really really like it. A book that totally did it for me is Essential Slick by Richard Dallaway and Jonathan Ferguson. Look it up. (No really!)

There are some things to Slick that, once understood, sort of reveals the whole pie in one go.

Queries produce results by executing actions onto the database. Both of those steps are asynchronous and depending on how you compose your actions, there may be a quirk or two around there too.

Patrik

On 26 March 2017, at 20:24, kraythe <[hidden email]> wrote:

Although I cannot use the Scala language at work because the productivity of our team would crater, I have still an interest in a language I would love to be able to use in a business world. Deciding that perhaps I was too harsh on the language, I have been playing with a couple of Scala libraries to try and give the language another shot in my off hours work. I just spent the better part of 7 hours playing with Slick 3.2.0 in Akka and Scala. I was massively frustrated and it turned out that yet again it is an implicit conversion that hosed me. Consider the following slick project.

Users.scala
package persistence.entities import slick.driver.H2Driver.api._ case class User(id: Long, name: String) class UserTable(tag: Tag) extends Table[User](tag, "user") { def id = column[Long]("id", O.PrimaryKey, O.AutoInc) def name = column[String]("name") def * = (id, name) <> (User.tupled, User.unapply) }

UsersDAL.scala
package persistence.dal import persistence.entities.{Tables, User, UserTable} import slick.jdbc.JdbcProfile import slick.lifted.Query import scala.concurrent.Future trait UserDAL { def findById(id: Long) : Future[Option[User]] } class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL { override def findById(userId: Long) = null def foo (userId : Long) = { val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId) db.run(filter.result) } }

Should be simple right? No. The db.run(filter.result) line does not compile on the result call. OK, time to comb the Slick documentation. No dice, my code looks just like theirs does fundamentally. So why is the result method not available. Do I have the wrong type? Nope. What about the wrong library? Nope. Update from 3.1.1 to 3.2.0 ? Done ... but ... no. What about the wrong structure for my table? Am I missing something? Comb example code ... and nope. OK, I am frustrated as all get out. In the meantime my irritation grows when I see example code like the following.

override def insert(rows: Seq[A]): Future[Seq[Long]] = { db.run(tableQ returning tableQ.map(_.id) ++= rows.filter(_.isValid)) }

Good god, how to decode that mess. Obviously run is taking something as a parameter but exactly what is the order of operations here? Anyway, I digress.

Back to the subject at hand, I am at a complete loss for how to explain the problem here so, I decide to create a post on the Slick forums to figure out what I am doing wrong. I think, for simplicity sake lets combine all of our classes into one file.

package persistence.entities import java.time.Instant import slick.driver.H2Driver.api._ import slick.jdbc.JdbcProfile import slick.lifted.{Query, TableQuery} import scala.concurrent.Future case class User(id: Long, name: String) class UserTable(tag: Tag) extends Table[User](tag, "user") { def id = column[Long]("id", O.PrimaryKey, O.AutoInc) def name = column[String]("name") def * = (id, name) <> (User.tupled, User.unapply) } object Users { val users = TableQuery[UserTable] trait UserDAL { def findById(id: Long) : Future[Option[User]] } class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL { override def findById(userId: Long) = null def foo (userId : Long) = { val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId) db.run(filter.result) } } }

OK, now that we have combined them we are ready to post this .. but wait ... it compiles now. What The Fork? Oh wait. There was an implicit conversion going on in there somewhere. There was something in the new file converting our query type into something that had the result method predefined. Magically, behind the scenes and with the right import it doesn't happen and the code doesn't compile.After experimentally deleting imports i discover it's import slick.driver.H2Driver.api._ that is the home of the implicit conversion. So I go look at the file and nope, no implicit. I drill and drill and finally find it. Wonderful. 7+ hours wasted for this. The fun part is the implict conversion happened as a result of the implicit conversion of an implicit conversion of an implicit conversion. Do the designers of Scala think this is actually good? You are just supposed to "know" that the conversion of a conversion of a conversion is happening I guess. Or is the goal to make the code so cryptic and opaque that the source of the library is entirely unreadable?

Given my original UsersDAL file there was no way for me to know that to make the code compile I had to find some implict conversion that could turn a Query into a DBIOAction. There is no means for you to know where this conversion is, what file has to be imported. All of the example code just assumes you must already know. Its fine copying the example code but if you modify, reorganize code into many files you break the delicately balanced Jenga tower and it doesn't work.

So after playing around for a few days with this project, I can see that if I ever brought this code into a business environment my CEO would absolutely murder me as it would take 6 months to add the most basic of features, not to mention 1+ years to train a new developer to be even moderately productive. Once again, its the implicit and its use in the language that is at the core of the problem. It actually makes me sad, but it would be wholesale irresponsible of me as a Principal Software Architect to recommend these techs in a business that makes its money putting out product for users in a short time.

So why post this? Because otherwise i LIKE Scala and wish it could be different. But its becoming like the JDK, they have decided that is the direction they are going and no one on the planet can convince them otherwise. OK now that I am done stating my opinion, I am ready to get flamed again. :-)

-- Robert Simmons Jr.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Rex Kerr-2

Re: Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

In reply to this post by kraythe

On Sun, Mar 26, 2017 at 11:24 AM, kraythe <[hidden email]> wrote:

I was massively frustrated and it turned out that yet again it is an implicit conversion that hosed me.

Maybe you would consider learning how to detect when implicit conversions are in play instead of assuming they're not and growing frustrated?

It seems like it's taking you a *very* long time to figure out what the issue is. And I've done the same thing, in multiple cases, and it *is* very frustrating, but when I realize that it's something that I could have recognized in advance and greatly sped up my search, I tend to redirect my frustration at myself (and/or view it as a learning opportunity) and if appropriate at the poor documentation of the library.

It takes about five minutes to figure out that an implicit is the issue. It might take a lot longer than that to figure out how to use it correctly, but it *that* it is a missing implicit shouldn't be mysterious. First, the error message will probably tell you, but I am not set up to compile Slick and you didn't share the message, so let's say you got some random "diverging implicit expansion" error that was pretty uninformative and comes up pretty often when your types are wrong.

Consider the following slick project.

Users.scala
package persistence.entities import slick.driver.H2Driver.api._ case class User(id: Long, name: String) class UserTable(tag: Tag) extends Table[User](tag, "user") { def id = column[Long]("id", O.PrimaryKey, O.AutoInc) def name = column[String]("name") def * = (id, name) <> (User.tupled, User.unapply) }

UsersDAL.scala
package persistence.dal import persistence.entities.{Tables, User, UserTable} import slick.jdbc.JdbcProfile import slick.lifted.Query import scala.concurrent.Future trait UserDAL { def findById(id: Long) : Future[Option[User]] } class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL { override def findById(userId: Long) = null def foo (userId : Long) = { val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId) db.run(filter.result) } }

Should be simple right? No. The db.run(filter.result) line does not compile on the result call. OK, time to comb the Slick documentation.

At which point you should realize two things, after no more than five minutes:

(1) `Query` doesn't have a `result` method

(2) `db.run` takes a `DBIOAction` type, which isn't a `Query`

So, the docs say the method isn't on the type you've got, and it returns another type that you also haven't got.

So there are two possibilities: `result` is an extension method, or the example isn't for the version you're using.

All right, where *is* the `result` method? The new Scaladoc (2.12) has a better search function, but the old Scaladoc has little letters under the search box in the top left, and if you click on R you get down to a line that looks like

result

QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl ConstArrayBuilder

Hm, `QueryActionExtensionMethodsImpl` sounds just like what we need, doesn't it?

The first one says it returns a JdbcDriver.DriverAction, which we follow and find is a FixedSqlAction which has DBIOAction as a supertype. Great! Now, how do we get `QueryActionExtensionMethodsImpl` to work?

At this point, you know to look for imports to get the method, and you can hopefully notice that working examples contain a `something.api._` import.

No dice, my code looks just like theirs does fundamentally. So why is the result method not available. Do I have the wrong type? Nope. What about the wrong library? Nope. Update from 3.1.1 to 3.2.0 ? Done ... but ... no. What about the wrong structure for my table? Am I missing something?

Again, if the above takes more than 5-10 minutes, you're allowing yourself to forget that Scala allows extension methods.

Comb example code ... and nope. OK, I am frustrated as all get out. In the meantime my irritation grows when I see example code like the following.

override def insert(rows: Seq[A]): Future[Seq[Long]] = { db.run(tableQ returning tableQ.map(_.id) ++= rows.filter(_.isValid)) }

Good god, how to decode that mess. Obviously run is taking something as a parameter but exactly what is the order of operations here?

Indeed. I wouldn't recommend that as idiomatic code even if it does compile, much like mixing || and && without parens. One probably has precedence over the other...but it's bad form to rely on it.

Oh wait. There was an implicit conversion going on in there somewhere. There was something in the new file converting our query type into something that had the result method predefined. Magically, behind the scenes and with the right import it doesn't happen and the code doesn't compile.

This attitude isn't going to help you or anyone else benefit from this feature.

You should be thinking:

Why isn't result just a method on the query?

What *is* the result of a query, anyway, and how do you get it?

Well, you have to actually contact the database.

Hm, are the details the same for all databases?

Probably not.

Ugh, why didn't they just make it take a parameter, or have it be a method on something that knows about database details?

All right, fine, there's got to be some glue.

After experimentally deleting imports i discover it's import slick.driver.H2Driver.api._ that is the home of the implicit conversion.

Since it's the only wildcard import, you can be pretty sure that's the one that's doing it.

So I go look at the file and nope, no implicit. I drill and drill and finally find it. Wonderful. 7+ hours wasted for this.

Except you didn't need to drill and drill unless you wanted to know exactly where it was coming from. At this point, you've solved your problem and now you're switching over from solving your problem to learning about how the library is structured. Nothing wrong with that, but it's worth keeping the two straight.

The fun part is the implict conversion happened as a result of the implicit conversion of an implicit conversion of an implicit conversion. Do the designers of Scala think this is actually good?

The question is whether the designers of Scala should make sure that the designers of Slick are unable to do this.

You are just supposed to "know" that the conversion of a conversion of a conversion is happening I guess. Or is the goal to make the code so cryptic and opaque that the source of the library is entirely unreadable?

At this point, it's the same problem as Java libraries where you have towers of abstract factory implementation builders etc.. It's almost impossible to find where the important stuff is actually happening because of how hard it is to tell at what level of abstraction the functionality you're using exists.

I have fits with big Java projects practically every time in this case, from Apache Commons Math (how the *#$&*#@% do you get your data into a form that this optimizer will use and how do you actually run the optimization?!) to Jackson to...well...everything. Big libraries often have huge towers of abstraction that make them impenetrable to a casual user.

That an implicit is involved somewhat complicates things, but not so much.

Given my original UsersDAL file there was no way for me to know that to make the code compile I had to find some implict conversion that could turn a Query into a DBIOAction.

That part should have been easy to figure out.

There is no means for you to know where this conversion is, what file has to be imported.

That part is *not* so easy to figure out if you're lacking working examples. If you *do* have working examples, you can spot where it must be happening even if you don't know what.

All of the example code just assumes you must already know.

Yeah, but I have exactly the same experience with big frameworky Java projects. If I don't use exactly the set of factories and so on that they do, in exactly the order they do, the types don't line up, it doesn't compile, and it's practically impossible to figure out how to get your hands on the actual instance you need. (Heck, even java.awt is like this if you start with pixels in memory and want an image.)

Anyway, I would not have chosen to structure Slick this way, mostly because I like knowing how the internals are working and this makes it hard to follow. But if you are perplexed by the existence of implicits at *this* point, after having apparently had so much grief with them, maybe you need to sit back and decide whether you're willing to learn how to work with implicits, or whether you want to keep being surprised by them.

Cause the latter isn't apparently very much fun.

(You can still find Slick not a joy to work with and want some other DB access library, of course.)

The penalty of removing this sort of implicit evidence, though, is that it is vastly harder to set up context that will make everything work the right way, and then forget about it as unimportant.

--Rex

Rex Kerr-2

Re: Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

It occurs to me that I was using the term "extension methods" extremely loosely here to cover "any kind of implicit conversion enabling invocation of a method that doesn't exist on the type it's called on".

Technically, Scala doesn't have extension methods at all, and implicit classes which serve the same purpose as a dedicated extension method feature are not the only way to get methods appearing--any implicit conversion can do it.

But the point is that if you see foo.bar and foo's type doesn't have a bar method on it, it shouldn't be a huge surprise.

--Rex

On Sun, Mar 26, 2017 at 5:14 PM, Rex Kerr <[hidden email]> wrote:

On Sun, Mar 26, 2017 at 11:24 AM, kraythe <[hidden email]> wrote:
I was massively frustrated and it turned out that yet again it is an implicit conversion that hosed me.

Maybe you would consider learning how to detect when implicit conversions are in play instead of assuming they're not and growing frustrated?

It seems like it's taking you a *very* long time to figure out what the issue is. And I've done the same thing, in multiple cases, and it *is* very frustrating, but when I realize that it's something that I could have recognized in advance and greatly sped up my search, I tend to redirect my frustration at myself (and/or view it as a learning opportunity) and if appropriate at the poor documentation of the library.

It takes about five minutes to figure out that an implicit is the issue. It might take a lot longer than that to figure out how to use it correctly, but it *that* it is a missing implicit shouldn't be mysterious. First, the error message will probably tell you, but I am not set up to compile Slick and you didn't share the message, so let's say you got some random "diverging implicit expansion" error that was pretty uninformative and comes up pretty often when your types are wrong.

Consider the following slick project.

Users.scala
package persistence.entities import slick.driver.H2Driver.api._ case class User(id: Long, name: String) class UserTable(tag: Tag) extends Table[User](tag, "user") { def id = column[Long]("id", O.PrimaryKey, O.AutoInc) def name = column[String]("name") def * = (id, name) <> (User.tupled, User.unapply) }

UsersDAL.scala
package persistence.dal import persistence.entities.{Tables, User, UserTable} import slick.jdbc.JdbcProfile import slick.lifted.Query import scala.concurrent.Future trait UserDAL { def findById(id: Long) : Future[Option[User]] } class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL { override def findById(userId: Long) = null def foo (userId : Long) = { val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId) db.run(filter.result) } }

Should be simple right? No. The db.run(filter.result) line does not compile on the result call. OK, time to comb the Slick documentation.

At which point you should realize two things, after no more than five minutes:

(1) `Query` doesn't have a `result` method
(2) `db.run` takes a `DBIOAction` type, which isn't a `Query`

So, the docs say the method isn't on the type you've got, and it returns another type that you also haven't got.

So there are two possibilities: `result` is an extension method, or the example isn't for the version you're using.

All right, where *is* the `result` method? The new Scaladoc (2.12) has a better search function, but the old Scaladoc has little letters under the search box in the top left, and if you click on R you get down to a line that looks like

result

QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl ConstArrayBuilder

Hm, `QueryActionExtensionMethodsImpl` sounds just like what we need, doesn't it?

The first one says it returns a JdbcDriver.DriverAction, which we follow and find is a FixedSqlAction which has DBIOAction as a supertype. Great! Now, how do we get `QueryActionExtensionMethodsImpl` to work?

At this point, you know to look for imports to get the method, and you can hopefully notice that working examples contain a `something.api._` import.

No dice, my code looks just like theirs does fundamentally. So why is the result method not available. Do I have the wrong type? Nope. What about the wrong library? Nope. Update from 3.1.1 to 3.2.0 ? Done ... but ... no. What about the wrong structure for my table? Am I missing something?

Again, if the above takes more than 5-10 minutes, you're allowing yourself to forget that Scala allows extension methods.

Comb example code ... and nope. OK, I am frustrated as all get out. In the meantime my irritation grows when I see example code like the following.

override def insert(rows: Seq[A]): Future[Seq[Long]] = { db.run(tableQ returning tableQ.map(_.id) ++= rows.filter(_.isValid)) }

Good god, how to decode that mess. Obviously run is taking something as a parameter but exactly what is the order of operations here?

Indeed. I wouldn't recommend that as idiomatic code even if it does compile, much like mixing || and && without parens. One probably has precedence over the other...but it's bad form to rely on it.

Oh wait. There was an implicit conversion going on in there somewhere. There was something in the new file converting our query type into something that had the result method predefined. Magically, behind the scenes and with the right import it doesn't happen and the code doesn't compile.

This attitude isn't going to help you or anyone else benefit from this feature.

You should be thinking:

Why isn't result just a method on the query?
What *is* the result of a query, anyway, and how do you get it?
Well, you have to actually contact the database.
Hm, are the details the same for all databases?
Probably not.
Ugh, why didn't they just make it take a parameter, or have it be a method on something that knows about database details?
All right, fine, there's got to be some glue.

After experimentally deleting imports i discover it's import slick.driver.H2Driver.api._ that is the home of the implicit conversion.

Since it's the only wildcard import, you can be pretty sure that's the one that's doing it.

So I go look at the file and nope, no implicit. I drill and drill and finally find it. Wonderful. 7+ hours wasted for this.

Except you didn't need to drill and drill unless you wanted to know exactly where it was coming from. At this point, you've solved your problem and now you're switching over from solving your problem to learning about how the library is structured. Nothing wrong with that, but it's worth keeping the two straight.

The fun part is the implict conversion happened as a result of the implicit conversion of an implicit conversion of an implicit conversion. Do the designers of Scala think this is actually good?

The question is whether the designers of Scala should make sure that the designers of Slick are unable to do this.

You are just supposed to "know" that the conversion of a conversion of a conversion is happening I guess. Or is the goal to make the code so cryptic and opaque that the source of the library is entirely unreadable?

At this point, it's the same problem as Java libraries where you have towers of abstract factory implementation builders etc.. It's almost impossible to find where the important stuff is actually happening because of how hard it is to tell at what level of abstraction the functionality you're using exists.

I have fits with big Java projects practically every time in this case, from Apache Commons Math (how the *#$&*#@% do you get your data into a form that this optimizer will use and how do you actually run the optimization?!) to Jackson to...well...everything. Big libraries often have huge towers of abstraction that make them impenetrable to a casual user.

That an implicit is involved somewhat complicates things, but not so much.

Given my original UsersDAL file there was no way for me to know that to make the code compile I had to find some implict conversion that could turn a Query into a DBIOAction.

That part should have been easy to figure out.

There is no means for you to know where this conversion is, what file has to be imported.

That part is *not* so easy to figure out if you're lacking working examples. If you *do* have working examples, you can spot where it must be happening even if you don't know what.

All of the example code just assumes you must already know.

Yeah, but I have exactly the same experience with big frameworky Java projects. If I don't use exactly the set of factories and so on that they do, in exactly the order they do, the types don't line up, it doesn't compile, and it's practically impossible to figure out how to get your hands on the actual instance you need. (Heck, even java.awt is like this if you start with pixels in memory and want an image.)

Anyway, I would not have chosen to structure Slick this way, mostly because I like knowing how the internals are working and this makes it hard to follow. But if you are perplexed by the existence of implicits at *this* point, after having apparently had so much grief with them, maybe you need to sit back and decide whether you're willing to learn how to work with implicits, or whether you want to keep being surprised by them.

Cause the latter isn't apparently very much fun.

(You can still find Slick not a joy to work with and want some other DB access library, of course.)

The penalty of removing this sort of implicit evidence, though, is that it is vastly harder to set up context that will make everything work the right way, and then forget about it as unimportant.

--Rex

Peter Wolf

Re: Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

In reply to this post by kraythe

I and my team love Scala and Slick for our business, but we empathise with your feelings about implicits.

We strongly discourage the use of implicits in our own code, and import them only when necessary (e.g. Futures and scala.concurrent.ExecutionContext.Implicits.global). We also went through all our code, ripped out all the scala.collection.JavaConversions, and replaced them with scala.collection.JavaConverters.

We don't like implicits because they are "magic"... They are invisible and change the behavior of the code.

But, luckily, you can (mostly) avoid implicits if you disapprove on them :-)

Naftoli Gugenheim

Re: Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

I like Slick a lot. But if you want to avoid some of the more advanced ways scala can be used, I would recommend using a different database library. I hear good things about ScalikeJDBC from people with that kind of preference.

On Mon, Mar 27, 2017 at 2:05 PM Peter Wolf <[hidden email]> wrote:

I and my team love Scala and Slick for our business, but we empathise with your feelings about implicits.

We strongly discourage the use of implicits in our own code, and import them only when necessary (e.g. Futures and scala.concurrent.ExecutionContext.Implicits.global). We also went through all our code, ripped out all the scala.collection.JavaConversions, and replaced them with scala.collection.JavaConverters.

We don't like implicits because they are "magic"... They are invisible and change the behavior of the code.

But, luckily, you can (mostly) avoid implicits if you disapprove on them :-)

P

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Naftoli Gugenheim

Re: Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

In reply to this post by kraythe

Perhaps have a look at what I wrote here: https://groups.google.com/d/msg/scala-user/atl2dHv1gMY/NeQVI_5KAgAJ

On Sun, Mar 26, 2017 at 2:24 PM kraythe <[hidden email]> wrote:

Although I cannot use the Scala language at work because the productivity of our team would crater, I have still an interest in a language I would love to be able to use in a business world. Deciding that perhaps I was too harsh on the language, I have been playing with a couple of Scala libraries to try and give the language another shot in my off hours work. I just spent the better part of 7 hours playing with Slick 3.2.0 in Akka and Scala. I was massively frustrated and it turned out that yet again it is an implicit conversion that hosed me. Consider the following slick project.

Users.scala
package persistence.entities import slick.driver.H2Driver.api._ case class User(id: Long, name: String) class UserTable(tag: Tag) extends Table[User](tag, "user") { def id = column[Long]("id", O.PrimaryKey, O.AutoInc) def name = column[String]("name") def * = (id, name) <> (User.tupled, User.unapply) }

UsersDAL.scala
package persistence.dal import persistence.entities.{Tables, User, UserTable} import slick.jdbc.JdbcProfile import slick.lifted.Query import scala.concurrent.Future trait UserDAL { def findById(id: Long) : Future[Option[User]] } class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL { override def findById(userId: Long) = null def foo (userId : Long) = { val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId) db.run(filter.result) } }

Should be simple right? No. The db.run(filter.result) line does not compile on the result call. OK, time to comb the Slick documentation. No dice, my code looks just like theirs does fundamentally. So why is the result method not available. Do I have the wrong type? Nope. What about the wrong library? Nope. Update from 3.1.1 to 3.2.0 ? Done ... but ... no. What about the wrong structure for my table? Am I missing something? Comb example code ... and nope. OK, I am frustrated as all get out. In the meantime my irritation grows when I see example code like the following.

override def insert(rows: Seq[A]): Future[Seq[Long]] = { db.run(tableQ returning tableQ.map(_.id) ++= rows.filter(_.isValid)) }

Good god, how to decode that mess. Obviously run is taking something as a parameter but exactly what is the order of operations here? Anyway, I digress.

Back to the subject at hand, I am at a complete loss for how to explain the problem here so, I decide to create a post on the Slick forums to figure out what I am doing wrong. I think, for simplicity sake lets combine all of our classes into one file.

package persistence.entities import java.time.Instant import slick.driver.H2Driver.api._ import slick.jdbc.JdbcProfile import slick.lifted.{Query, TableQuery} import scala.concurrent.Future case class User(id: Long, name: String) class UserTable(tag: Tag) extends Table[User](tag, "user") { def id = column[Long]("id", O.PrimaryKey, O.AutoInc) def name = column[String]("name") def * = (id, name) <> (User.tupled, User.unapply) } object Users { val users = TableQuery[UserTable] trait UserDAL { def findById(id: Long) : Future[Option[User]] } class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL { override def findById(userId: Long) = null def foo (userId : Long) = { val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId) db.run(filter.result) } } }

OK, now that we have combined them we are ready to post this .. but wait ... it compiles now. What The Fork? Oh wait. There was an implicit conversion going on in there somewhere. There was something in the new file converting our query type into something that had the result method predefined. Magically, behind the scenes and with the right import it doesn't happen and the code doesn't compile.After experimentally deleting imports i discover it's import slick.driver.H2Driver.api._ that is the home of the implicit conversion. So I go look at the file and nope, no implicit. I drill and drill and finally find it. Wonderful. 7+ hours wasted for this. The fun part is the implict conversion happened as a result of the implicit conversion of an implicit conversion of an implicit conversion. Do the designers of Scala think this is actually good? You are just supposed to "know" that the conversion of a conversion of a conversion is happening I guess. Or is the goal to make the code so cryptic and opaque that the source of the library is entirely unreadable?

Given my original UsersDAL file there was no way for me to know that to make the code compile I had to find some implict conversion that could turn a Query into a DBIOAction. There is no means for you to know where this conversion is, what file has to be imported. All of the example code just assumes you must already know. Its fine copying the example code but if you modify, reorganize code into many files you break the delicately balanced Jenga tower and it doesn't work.

So after playing around for a few days with this project, I can see that if I ever brought this code into a business environment my CEO would absolutely murder me as it would take 6 months to add the most basic of features, not to mention 1+ years to train a new developer to be even moderately productive. Once again, its the implicit and its use in the language that is at the core of the problem. It actually makes me sad, but it would be wholesale irresponsible of me as a Principal Software Architect to recommend these techs in a business that makes its money putting out product for users in a short time.

So why post this? Because otherwise i LIKE Scala and wish it could be different. But its becoming like the JDK, they have decided that is the direction they are going and no one on the planet can convince them otherwise. OK now that I am done stating my opinion, I am ready to get flamed again. :-)

-- Robert Simmons Jr.

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Martijn Hoekstra

Re: Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala.

In reply to this post by Rex Kerr-2

On Mon, Mar 27, 2017 at 2:14 AM, Rex Kerr <[hidden email]> wrote:

On Sun, Mar 26, 2017 at 11:24 AM, kraythe <[hidden email]> wrote:
I was massively frustrated and it turned out that yet again it is an implicit conversion that hosed me.

Maybe you would consider learning how to detect when implicit conversions are in play instead of assuming they're not and growing frustrated?

It seems like it's taking you a *very* long time to figure out what the issue is. And I've done the same thing, in multiple cases, and it *is* very frustrating, but when I realize that it's something that I could have recognized in advance and greatly sped up my search, I tend to redirect my frustration at myself (and/or view it as a learning opportunity) and if appropriate at the poor documentation of the library.

It takes about five minutes to figure out that an implicit is the issue. It might take a lot longer than that to figure out how to use it correctly, but it *that* it is a missing implicit shouldn't be mysterious. First, the error message will probably tell you, but I am not set up to compile Slick and you didn't share the message, so let's say you got some random "diverging implicit expansion" error that was pretty uninformative and comes up pretty often when your types are wrong.

Consider the following slick project.

Users.scala
package persistence.entities import slick.driver.H2Driver.api._ case class User(id: Long, name: String) class UserTable(tag: Tag) extends Table[User](tag, "user") { def id = column[Long]("id", O.PrimaryKey, O.AutoInc) def name = column[String]("name") def * = (id, name) <> (User.tupled, User.unapply) }

UsersDAL.scala
package persistence.dal import persistence.entities.{Tables, User, UserTable} import slick.jdbc.JdbcProfile import slick.lifted.Query import scala.concurrent.Future trait UserDAL { def findById(id: Long) : Future[Option[User]] } class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL { override def findById(userId: Long) = null def foo (userId : Long) = { val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId) db.run(filter.result) } }

Should be simple right? No. The db.run(filter.result) line does not compile on the result call. OK, time to comb the Slick documentation.

At which point you should realize two things, after no more than five minutes:

(1) `Query` doesn't have a `result` method
(2) `db.run` takes a `DBIOAction` type, which isn't a `Query`

So, the docs say the method isn't on the type you've got, and it returns another type that you also haven't got.

So there are two possibilities: `result` is an extension method, or the example isn't for the version you're using.

All right, where *is* the `result` method? The new Scaladoc (2.12) has a better search function, but the old Scaladoc has little letters under the search box in the top left, and if you click on R you get down to a line that looks like

result

QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl QueryActionExtensionMethodsImpl StreamingQueryActionExtensionMethodsImpl ConstArrayBuilder

Hm, `QueryActionExtensionMethodsImpl` sounds just like what we need, doesn't it?

The first one says it returns a JdbcDriver.DriverAction, which we follow and find is a FixedSqlAction which has DBIOAction as a supertype. Great! Now, how do we get `QueryActionExtensionMethodsImpl` to work?

At this point, you know to look for imports to get the method, and you can hopefully notice that working examples contain a `something.api._` import.

No dice, my code looks just like theirs does fundamentally. So why is the result method not available. Do I have the wrong type? Nope. What about the wrong library? Nope. Update from 3.1.1 to 3.2.0 ? Done ... but ... no. What about the wrong structure for my table? Am I missing something?

Again, if the above takes more than 5-10 minutes, you're allowing yourself to forget that Scala allows extension methods.

Comb example code ... and nope. OK, I am frustrated as all get out. In the meantime my irritation grows when I see example code like the following.

override def insert(rows: Seq[A]): Future[Seq[Long]] = { db.run(tableQ returning tableQ.map(_.id) ++= rows.filter(_.isValid)) }

Good god, how to decode that mess. Obviously run is taking something as a parameter but exactly what is the order of operations here?

Indeed. I wouldn't recommend that as idiomatic code even if it does compile, much like mixing || and && without parens. One probably has precedence over the other...but it's bad form to rely on it.

Oh wait. There was an implicit conversion going on in there somewhere. There was something in the new file converting our query type into something that had the result method predefined. Magically, behind the scenes and with the right import it doesn't happen and the code doesn't compile.

This attitude isn't going to help you or anyone else benefit from this feature.

You should be thinking:

Why isn't result just a method on the query?
What *is* the result of a query, anyway, and how do you get it?
Well, you have to actually contact the database.
Hm, are the details the same for all databases?
Probably not.
Ugh, why didn't they just make it take a parameter, or have it be a method on something that knows about database details?
All right, fine, there's got to be some glue.

After experimentally deleting imports i discover it's import slick.driver.H2Driver.api._ that is the home of the implicit conversion.

Since it's the only wildcard import, you can be pretty sure that's the one that's doing it.

This part is what I quite often struggle with when exploring a new library: Wildcard imports in example code, not only, but especially those that provide implicits. Or even worse, example code without imports.

Small self-contained examples would be much more helpful to the way I learn if the imports don't have wildcards, and comments help a lot.

import slick.driver.H2Driver.api._ // provide implicit conversions for Query(, Foo, Blarp and Quux)

would already be very helpful.

That's more of a documentation issue (and probably an issue of what different people find helpful, I'm not sure if I'm representative for this) than an implicits issue though.

So I go look at the file and nope, no implicit. I drill and drill and finally find it. Wonderful. 7+ hours wasted for this.

Except you didn't need to drill and drill unless you wanted to know exactly where it was coming from. At this point, you've solved your problem and now you're switching over from solving your problem to learning about how the library is structured. Nothing wrong with that, but it's worth keeping the two straight.

The fun part is the implict conversion happened as a result of the implicit conversion of an implicit conversion of an implicit conversion. Do the designers of Scala think this is actually good?

The question is whether the designers of Scala should make sure that the designers of Slick are unable to do this.

You are just supposed to "know" that the conversion of a conversion of a conversion is happening I guess. Or is the goal to make the code so cryptic and opaque that the source of the library is entirely unreadable?

At this point, it's the same problem as Java libraries where you have towers of abstract factory implementation builders etc.. It's almost impossible to find where the important stuff is actually happening because of how hard it is to tell at what level of abstraction the functionality you're using exists.

I have fits with big Java projects practically every time in this case, from Apache Commons Math (how the *#$&*#@% do you get your data into a form that this optimizer will use and how do you actually run the optimization?!) to Jackson to...well...everything. Big libraries often have huge towers of abstraction that make them impenetrable to a casual user.

That an implicit is involved somewhat complicates things, but not so much.

Given my original UsersDAL file there was no way for me to know that to make the code compile I had to find some implict conversion that could turn a Query into a DBIOAction.

That part should have been easy to figure out.

There is no means for you to know where this conversion is, what file has to be imported.

That part is *not* so easy to figure out if you're lacking working examples. If you *do* have working examples, you can spot where it must be happening even if you don't know what.

All of the example code just assumes you must already know.

Yeah, but I have exactly the same experience with big frameworky Java projects. If I don't use exactly the set of factories and so on that they do, in exactly the order they do, the types don't line up, it doesn't compile, and it's practically impossible to figure out how to get your hands on the actual instance you need. (Heck, even java.awt is like this if you start with pixels in memory and want an image.)

Anyway, I would not have chosen to structure Slick this way, mostly because I like knowing how the internals are working and this makes it hard to follow. But if you are perplexed by the existence of implicits at *this* point, after having apparently had so much grief with them, maybe you need to sit back and decide whether you're willing to learn how to work with implicits, or whether you want to keep being surprised by them.

Cause the latter isn't apparently very much fun.

(You can still find Slick not a joy to work with and want some other DB access library, of course.)

The penalty of removing this sort of implicit evidence, though, is that it is vastly harder to set up context that will make everything work the right way, and then forget about it as unimportant.

--Rex

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.