Scala combines a lot of excellent features (functional-style pattern matching, an expressive type system, closures, etc.) with JVM compatibility and a very interesting developer ecosystem (e.g., Akka, Play, Lift, scalacheck, and Spark, just to name a few notable projects). Fedora has included a package for Scala itself for some time, but it doesn’t include any of the ecosystem projects. The main obstacle to having Scala ecosystem projects in Fedora is that many projects use sbt, the Simple Build Tool, but there is no native Fedora sbt package. In this post, I’m going to discuss some of the things that make sbt very interesting as a build tool but challenging to package for Fedora, as well as the solutions I’ve come up with to these problems. First, however, we’ll discuss some background.
I’ve noted in the past that there is a big tension between the Fedora model of dependency management and the models adopted by many language-specific dependency managers. Put simply, the Fedora model is that projects should depend upon system copies of the latest versions of libraries, which have been built with system tools, which were themselves built from pristine sources in a controlled environment. Language-specific models, such as the ones we see with RubyGems and rvm; Python eggs; Java projects using Maven or Ivy; and Erlang releases (especially those managed with rebar), typically allow developers more flexibility to install multiple versions of libraries, fetch dependencies from canonical locations on the web or from source repositories, and rely on different versions of language environments and language runtimes. sbt, which provides both build and dependency management, is no exception in this regard; in fact, it provides as much flexibility as any other language-specific build tool I’ve encountered.
Launching
You don’t actually download sbt. Instead, you download a small, self-contained JAR file that will run on any Java 1.6 JRE and includes enough of sbt, Apache Ivy, and the Scala standard library to fetch the whole Scala standard library and compiler, sbt itself, and its dependencies. It can also fetch multiple versions of each of these. This approach means that it’s absolutely straightforward to get started using sbt in almost any environment with a JVM, but it conflicts with Fedora policies on bundling, single versions of libraries, and pristine sources.
My solution to this problem is to develop a Fedora-specific sbt launcher that is willing to run against system copies of sbt itself, the Scala compiler and libraries, and other locally-installed JAR files.
Dependency management
sbt uses Apache Ivy to manage dependencies. Fedora has excellent support for building packages that use Maven, but Ivy is still not well-represented in Fedora. Just as with Maven, most of the concerns that Ivy is meant to handle are either addressed by RPM itself (specifying versions of dependencies, finding transitive dependencies, etc.) or do not apply to packages that meet Fedora guidelines (e.g. running different projects against different versions of their dependencies).
It is possible (but clearly suboptimal) to build Ivy packages against RPM-installed dependencies by specifying an Ivy resolver pattern that ignores version numbers and finds JAR artifacts where Fedora packages put them in the filesystem, like this:
/usr/share/java/[artifact].[ext]However, /usr/share/java isn’t set up as a proper Ivy repository; it contains no Ivy module descriptor files (i.e., ivy.xml files). This isn’t a problem if we’re using Ivy from Ant or standalone, but sbt calls out to Ivy in a way that requires module descriptors and doesn’t expose the setting to make them optional.
I have solved this problem in two ways: the first is a simple script that makes an ersatz Ivy repository from locally-installed packages, which can then be used by an sbt build. The second is a small patch to sbt that exposes the Ivy setting to make module descriptor files optional. (I use the former to build sbt binaries that include the latter.)
Bootstrapping
sbt is used to build itself, as well as some of its dependencies. Fedora has a policy for packaging projects that need to bootstrap in this way, and some other build tools (like rebar) also depend on libraries that are built with that tool. Because of how sbt uses a launcher and because of its dependency management, it is trickier to bootstrap in Fedora than other similar projects (since the initial sbt binary must run locally and must incorporate other Fedora-specific patches, like the module descriptor patch above and patches to work with versions of libraries that ship in Fedora).
Where from here?
Having sbt in Fedora would remove the biggest barrier to getting a lot of the Scala ecosystem in to Fedora, and sbt is a really interesting framework in its own right. However, it’s one of the projects where the mismatch between what Fedora requires of upstream projects and the assumptions that contemporary developers work under is particularly pronounced. These difficulties aren’t insurmountable, although I found the way that they combine and interweave somewhat daunting when I started investigating sbt.
I’m envisioning a sbt package for Fedora that provides the best of both worlds: an unrestricted sbt environment for developers who want to use Fedora but have the flexibility to target development to other Scala versions (or to use libraries that are available in Ivy repositories but not in Fedora) and a Fedora-specific sbt script that builds software against system packages in a Fedora-friendly way, much like the xmvn and mvn-rpmbuild tools were for Maven. This way, Fedora packagers would have a straightforward way to generate high-quality RPMs from Scala sources and Scala hackers who just want Fedora to meet their needs today could use the system package without restrictions (while having a path to package their projects for Fedora in the future should they choose to).
I welcome feedback and collaboration from Scala hackers who’d like to use Fedora (or other downstream distributions with similar packaging constraints) and from Fedora hackers who’d like to see Fedora as a better place for Scala.