The pain of dependency management (that's you, vscaladoc-1.1-md-3)...

Well, it's certainly been a while since I've posted anything here, but I figured this is as good a place as any for this tidbit. While working on a project at work I ran "sbt clean-cache" accidentally, which ended up clearing out my lib_managed dir. "No problem," I thought, "I'll just run sbt update and make it right". Well, long story short, update failed to download vscaladoc-1.1-md-3:

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.scala-tools#vscaladoc;1.1-md-3: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

Wait, what? That's not one of my dependencies, it must be a transitive dependency from something else that's gumming things up. After some searching through my ~/.ivy directory for "1.1-md-3", I found the culprit: configgy. I'm not sure why configgy is exporting a dependency on vscaladoc (or on specs, for that matter), but I'm guessing that this is just the default behavios and Configgy needs fixed. Anyways, changing my SBT dep to:

val configgy = "net.lag" % "configgy" % "2.0.0" intransitive()

gets things going again by telling SBT to just ignore all of Configgy's upstream deps.

Amateur tools for professional works

Tyler (one of my co-authors on DGL) sent this link to me this week and it made me realize that I had some things to say as well.

First off, I agree 100% with Peter's assessment of the Royalty Statement. It reads like a tax form and it took me several passes to really figure out what was going on. The previous statements make a lot more sense. I can't fathom why they would want to make the statements more difficult to read, but who knows. It seems like we've really had to push hard to get any kind of numbers out of APress, almost like they're trying to hide things from us. So far Tyler and Marius haven't received their statements, even though it's more than a week since the statement should have arrived according to the contracts.

My second major complaint about the process was that we had to use Word to write the book (OpenOffice, in my case). At the time that we were negotiating with APress on the book, we were probably 60% done with it already, but it was written in LaTeX. Some of the other publishers that we were talking to either said that LaTeX was no problem or that they had previously helped people translate work from LaTeX. This was actually our main concern at the time, but we had such a positive recommendation on APress from someone else that we decided that this wasn't such a big deal. In hindsight, it was huge. LaTeX is designed to write large, structured documents and make it easy for the writer to do so. Word is not, and it would be extremely difficult for someone to convince me of this after our experience here.

Word, as a WYSIWYG editor, forces me to not only be a writer, but to be a typesetter and layout artist. That's fine if you want to write a one or two page letter, but for a 2-300 page book it was painful. Styles do little to alleviate the pain. Cross-referencing and indexing are also trivial with LaTeX, but not so simple in Word (or OO). Another thing that I loved about LaTeX that I missed in Word/OO was that LaTeX has a listings package that syntax highlights code for you. We just defined Scala's keywords and syntax in the LaTeX preamble and all of our listings looked very nice. This wasn't supported by APress' style sheets, and even if it was I don't know of any software that would do this for you. I don't think there's any way for me to adequately explain my frustration with using the wrong tool for the job here.

Related to this is APress' stylistic decision to not use numbered sections or page numbers in references. Instead, it was decided that when we need to reference a section, we should simply use the chapter or section name. This was really hard for me to swallow, especially since the LaTeX version does section and page numbered references just about automatically. That means that whenever we would add a listing or move an existing one, we had to hand-renumber them. It also means that text reads like:

Additionally, we use Lift's binding mechanism (see the "Binding Values in Snippets" Section)

instead of

Additionally, we use Lift’s binding mechanism (Section 3.11.1)...

Oh, and in the LaTeX PDF, those references are hyperlinks to take you directly to the right place. No such luck in DocWorld.

My third complaint is that there was little downstream information flow from APress in terms of editorial decisions. We found out after the fact that the title was changed from "Exploring Lift" to "The Definitive Guide to Lift". We also found out when we were nearing completion that there would be no index at the back of the book, something that I find incomprehensible for a tech book (this is also something that's trivial to do with LaTeX). Finally, as we neared the publishing date we were told that the appendices would no longer fit in the print version. The appendices are well referenced from the text so this was not a trivial change to the book. In the end we compromised by making all of the appendices available for free here:

http://www.apress.com/book/downloadfile/4390

There was supposed to be a page at the back of the book explaining that due to printing limitations the appendices had to be placed online, and giving that link, but somehow that page never made it into the book. I'll take some blame on that for not properly proofing the final draft, but it was embarrassing and frustrating when the book came out and most of the questions on the mailing list were "Where are the appendices?"

In the end it was a learning experience, just like it was for Peter. I wouldn't go so far as to say it was a bad experience, as it was definitely a pleasure (for me, at least) to work with the copy editors and other people in the production process at APress. I think that mostly I hope that this post will serve others who are looking to write books that they need to really carefully consider all of the aspects of what goes into writing a book, and what to look out for. I think I would have been happier with the process if I felt like there were less surprises. I also think that if I have to write another book I won't use Word. I think it would be far simpler to fake the same look using LaTeX class and style files than to go through that headache again.

One final note. Unlike Peter, we already had our work licensed under the Creative Commons before we signed on with APress. Because of that, I still maintain the open source version of the book here:

http://groups.google.com/group/the-lift-book

My complaints here notwithstanding, the APress version is a good book, and if you want to get started in Lift please check it out.

ScalaJPA hits 1.0

I've been working on packaging up some of the Scala <-> JPA boilerplate in use for the Lift JPA Demo into a small project called ScalaJPA. After some work on documentation and bringing it up to Lift 1.0 goodness, you can find the project here:

http://scala-tools.org/mvnsites/scalajpa/

Scaladoc is here:

http://scala-tools.org/mvnsites/scalajpa/scaladocs/index.html

If you're using Maven, simply drop this dependency into your pom.xml:

<dependency>
  <groupId>org.scala-libs>/groupId>
  <artifactId>scalajpa>/artifactId>
  <version>1.0>/version>
</dependency>

Just to give you an idea of how easy it is to use, here's how we would define a per-request EntityManager in Lift:

package com.foo.jpa

import org.scala_libs.scalajpa.{LocalEM,RequestVarEM}

object Model extends LocalEM("MyEMName") with RequestVarEM

The Model object will then act as a ScalaEntityManager that uses a new per-request EM for its operations. We can use it in our code like

...
val author = new Author("Kurt Vonnegut")
val book = new Book("Cat's Cradle", author)
Model.persistAndFlush(book)

In addition to the RequestVarEM, there are classes to handle a ThreadLocal EM as well as plain old factory classes that will let you set up and close your own ScalaEntityManagers. Check out the Lift JPA Demo site for more example code:

http://github.com/dpp/liftweb/tree/794cac5abf6b1ae5502f6321847f6186fcd8de90/sites/JPADemo

GRUB and arrow keys

OK, not really related to coding but I thought it would be useful to put this out there in case anyone runs into the same issue. I'm trying to use a remote VMWare console for a linux VM I'm running at work and for some reason the arrow keys won't work. That means that I can't move up and down to select the recovery boot. I finally figured out, after much trial and error, that you can use Ctrl-P for "up" and Ctrl-N for "down". Good luck!

More progress on JPA

I've refactored my code now that I have a request-per-session. So far, this is what the Model object looks like:

object Model extends Logger {
  def logname = "Model"

  lazy val factory = Persistence.createEntityManagerFactory("foo")

  // Per-request entity manager
  //object emVar extends RequestVar[EntityManager](null)
  //def em = emVar.is

  // Temporarily using ThreadLocal until we get lifecycle handling in RequestVar
  val emVar = new ThreadLocal[EntityManager]
  def em = emVar.get()

  /**
   This method allows me to clean up my code a bit and only handle JPA-related exceptions.
   An example usage would be:

   def addFood(newFood : Food) = 
     wrapEM({
       em.persist(newFood)
       S.redirectTo("/food/list.html")
     }, {
       case cve : ConstraintViolationException => S.error("That food already exists!")
       case _ => S.error("Internal error adding food")
     })

   Note that if I used normal try/catch then the wildcard match would trap the RedirectException
   thrown by S.redirectTo.
  */
  def wrapEM[A](f : => A, handler : PartialFunction[Throwable, A]) : A = {
    try {
      val tx = em.getTransaction()

      tx.begin()
      try {
        val ret : A = f
        ret
      } finally {
        // make sure that we commit even with a redirectexception
        tx.commit()
      }
    } catch {
      // Special case. Usually we want to know why it failed to commit, not just that it failed
      case re : RollbackException => {
        val (cause,message) = if (re.getCause() == null) { 
          (re,"No cause") 
        } else { 
          (re.getCause(), re.getCause().getMessage())
        }
        this.error("EM Commit error: {}", message)
        handler(cause)
      }
      case he : HibernateException => {
        this.error("Hibernate error", he)
        handler(he)
      }
      case pe : PersistenceException => {
        this.error("EM Error", pe)
        handler(pe)
      }
    }
  } 

  /**
   Bridging utility method. This should eventually go away.
   */
  
  def usingEM[A](f: EntityManager => Can[A]) : Can[A] = wrapEM(
    {f(em)}, {
      case e : Exception => Failure(e.getMessage(), Full(e), Nil)
    })

  /**
   Loads the given object and returns a Full can with the object if it could be loaded, Empty if it's not found.
   */
  def load[A](id : java.lang.Long, clazz : Class[A]) : Can[A] = load(this.em, id, clazz)
  def load[A](em : EntityManager, id : java.lang.Long, clazz : Class[A]) : Can[A] = em.find(clazz, id) match {
    case found : A => Full(found)
    case null => Empty
  }
  
  /**
    Queries for a single instance of an object based on the queryName and params.
    
    @return Full(x) if a single result is returned, Empty if no results are returned, Failure on any errors
    
   */
  def find[A](providedEM : EntityManager, queryName : String, params : Pair[String,Any]*) : Can[A] = {
    try {
      val query = providedEM.createNamedQuery(queryName)
      
      params foreach { param => query.setParameter(param._1, param._2) }
      
      Full(query.getSingleResult().asInstanceOf[A])
    } catch {
      case e: NoResultException => Empty
    }
  }

  /**
    Queries for a result list of objects based on the queryName and params

    @return A scala.collection.jcl.Buffer of results
   */
  def findAll[A](em : EntityManager, queryName : String, params : Pair[String,Any]*) : Buffer[A] = { 
    val query = em.createNamedQuery(queryName)
    
    params foreach { param => query.setParameter(param._1, param._2) }
      
    val result = query.getResultList().asInstanceOf[java.util.List[A]]
    
    new BufferWrapper[A] { override def underlying = result }
  }
  
  // Implicit defs to help with entity member access
  implicit def setToWrapper[A](set : java.util.Set[A]) = new SetWrapper[A]{override def underlying = set}
  implicit def listToWrapper[A](list : java.util.List[A]) = new BufferWrapper[A]{override def underlying = list}
}

I just found out that I can do JNDI and JTA in Jetty, so I might take a crack at getting that working once I've ripped out all of the old JPA access code in my various classes.

Evolution of JPA from Lift

Edit: Forgot that I could use RequestVar. Also, Look at my next post for a further evolution of the Model object

Well, with some suggestions and info from David and Viktor I've started overhauling my use of JPA from within Lift. First and foremost, I've added in support for an EntityManager-per-request. This requires a little setup and utility class in my case:

object Model {
  def createEM() = // fill this in however you want: JNDI, local Persistence, etc

  // functions for setting a per-request entity manager
  object emVar extends RequestVar[EntityManager](null)
  def em = emVar.is

  def wrapEM(f : => Unit, msg : String) : Unit = {
    try {
      f
    } catch { 
      case e : Exception => this.error("Unexpected exception", e); S.error(msg)
    }
  }
}

The wrapEM method is something I've put together to make my code a little more concise when all I want to do is display an error when something goes wrong. The real meat of EM-per-session happens in Boot:

// Set up a LoanWrapper to automatically instantiate and tear down the EntityManager on a per-request basis
S.addAround(List(
  new LoanWrapper { 
    def apply[T] (f : => T): T = {
      val em = Model.createEM()

      // Add EM into S scope
      Model.emVar(em)
   
      try {
        f
      } finally {
        em.close()
      }
    }
  }
))

JPA From Lift

This has been discussed a little on the lift mailing list so I thought I'd write something up. Lift has a nice, built-in mapper framework for doing database-object mapping, but it's geared mainly toward small, lightweight schemas. When you start to throw a lot of tables at it, or if you need to do legacy or out-of-the-mainstream work, you can start to run into limitations or concept mistmatches that can end up making more sophisticated (and heavyweight) frameworks look more attractive. In my case I have an existing Java project that has its own persistence layer using JPA (Hibernate EntityManager), so it really seemed like it would be best to reuse the existing code rather than trying to forklift the entire thing.

To start with, accessing JPA entities from Scala is really no different that accessing them from Java. In either case you're getting a JPA EntityManager interface from somewhere (JNDI in my case) and using the persist, merge, etc. methods to control object persistence management. There are, however, a few things you can do with Scala to make it a little simpler to use and more Scala-ish.

The first issue I came to was that all of my JPA entities use Java collections for their collection interfaces. In my case they're all java.util.Sets, but it's nicer to have a Scala set to get access to all of the nice methods like exists, toList, etc. For that, I start out with an implicit definition:

implicit def utilSetToSet[A] (s : java.util.Set[A]) =
 new SetWrapper[A] { override def underlying = s}

That way, when I access a set member of a given entity, I can automatically use it as a Scala set.

The next thing I've added to make things a little easier is to create a "using" method to automatically handle closing the entity manager and handling any exceptions thrown:

def usingEM[A](f : EntityManager => Can[A]) : Can[A] = {
  val em = emf.createEntityManager()

  try {
    f(em)
  } catch {
    case e: Exception => Failure(e.getMessage(), Full(e), Nil)
  } finally {
    em.close()
  }
}

I've been vacillating on this particular method, since in the past I usually have used a filter to open an EntityManager and then close it at the end. Doing it this way allows me to better handle errors via Scala's nice pattern matching, but it also makes the code more "baroque":

object entityName extends RequestVar[Can[String]](Empty)
def view (xhtml : NodeSeq) : NodeSeq = usingEM({em =>
    val myEntity = em.createNamedQuery("findEntityByName")
                     .setParamater("name", entityName.is openOr "non-existant")
                     .getSingleResult()

    Full(bind("me", xhtml, // blah blah))
  }) match {
    case Failure(_, e : NoResultException, _) => S.error("Could not find entity by name")
    case Failure(msg, exception, _) => log(msg, exception); S.error("Error retrieving entity"); Text("")
    case Full(x) => x
  }

I'm not convinced that this is the right way to do this so I'm open to suggestions.

Derek