opam2debian, a tool to create Debian binary package out of OPAM

One week ago, a thread in opam-devel mailing list started about the possibility to create binary snapshot to distribute OPAM packages. Distributing binary packages with everything already compiled is pretty useful when you want to make sure that everyone has the same version of packages installed and you don't want to spend time configuring all the computer of your colleague.

As a matter of fact I was also interested to see that happening. I have several computers where I want to install a set of packages and I want to snapshot OPAM archives when I am ready for an upgrade. I have tested for a long time another source distribution: GODI. I even wrote a puppet module to drive it. This was a fun experience, but as with any source only distribution, there are some drawbacks. Especially even if it is fully automated, you get a lot of errors and timeout when trying to build automatically from source.

This is how opam2debian has started. I wanted a replacement for puppet-godi that takes advantage of OPAM and the possibility to distribute my set of packages everywhere quickly.

The goals of this project are:

  1. generate a Debian binary package with all the dependencies set on external packages (like libpcre)
  2. use OPAM to build everything
  3. use standard Debian process to build package
  4. create snapshot of OPAM repository that allow to rebuild exactly the same thing on different arches

The real challenge behind opam2debian was to be able to use standard Debian process to create package to get all the power of dependencies computation provided by the Debian maintainer scripts. My great achievement on this topic was to find a tool called proot that allows you to bind mount a directory as a user. This is an achievement because it allows to create a package using a directory where you are not allowed to write (i.e. /opt/opam2debian directory which is owned by root). This is a way to do it on any Jenkins builder without root access and a standard way for Debian to build packages.

Usage example

I need to compile a set of package to install on all my Jenkins builder. Here is the list of packages I need and I also want to use the latest OCaml version. I want to build a Debian package for Debian Wheezy which has only OCaml 3.12.1.

Building the package:

$ opam2debian create --build --name opam2debian-test2 --compiler 4.01.0 \
 ocamlfind fileutils ocaml-data-notation ocaml-expect ounit \
 ocamlmod ocamlify oasis yojson sexplib extlib pcre-ocaml \
 calendar ocaml-inifiles ocamlnet ocurl gettext inotify ocaml-sqlexpr \
 ocamlrss ocaml-xdg-basedir

Getting the package list right can be tricky because the process will stop at the first error -- and if it is because the last package fails to build it can be long. I have implemented a --keep-build-dir in case you want to tune the package list live.

The program will build everything and it can takes quite a while. At the end you get a file opam2debian-test2_20131105_amd64.deb which has a reasonable size of 232MB. That is big, but it looks like the biggest directory is $OPAMROOT/4.01.0/build, I am not sure if I can remove it but we may save some space here (the build directory represents 50% of the package size).

Then you can just do a standard Debian installation:

$ sudo dpkg -i opam2debian-test2_20131105_amd64.deb

And you can use it:

$ eval $(opam config env --root /opt/opam2debian/opam2debian-test2/)
$ which ocamlfind
 /opt/opam2debian/opam2debian-test2/4.01.0/bin/ocamlfind
$ ocamlfind list
 [...]
 stdlib              (version: [distributed with Ocaml])
 str                 (version: [distributed with Ocaml])
 threads             (version: [distributed with Ocaml])
 threads.posix       (version: [internal])
 threads.vm          (version: [internal])
 type_conv           (version: 109.41.00)
 unidiff             (version: 0.0.2)
 unix                (version: [distributed with Ocaml])
 userconf            (version: 0.3.1)
 xdg-basedir         (version: 0.0.3)
 xmlm                (version: 1.1.1)
 yojson              (version: 1.1.5)

A nice thing to note, is that it is a standard OPAM install. You can install, update and upgrade OPAM from there, as root. However, I would recommend not to do it, and just rebuild a newer Debian package to upgrade.

Install

You will need the opam and proot Debian package available in Debian jessie and sid. You will also need various OCaml libraries (cmdliner, calendar, fileutils and jingoo).

Download the opam2debian tarball on the forge, build and install it.

The project is hosted on github.

Open issues

On the initial list of goals of the project. Not everything is completed. I have still several open issues.

In particular 4. (create snapshot of OPAM repository) was blocked by a bug in opam-mk-repo, that prevents snapshots (see my pull-request to solve it).

Another issues is about licenses of the included files. I should list them all and I need to figure out a way to extract every licences.

Submit bugs directly to github.

They posted on the same topic

Trackback URL : http://sylvain.le-gall.net/blog/index.php?trackback/86

This post's comments feed