Blog of Sylvain Le Gall

The new project shares the same code base, but this branch is a total rewrite of the project children.

Clementine is born on 8th July 2010, at 12:01 Paris time. She is now at home with her mother, father and brother. She is very peaceful and hardly cry once or twice in a day: at bath time and around 5am for her night lunch.

I will probably lack of time in the coming month. Expect some delays with the OCaml forge, other OCaml projects I maintain and my OCaml Debian packages.

This blog post is a kind of followup of a previous blog post about reading 10 years old CD-R.

Today, I get a new shiny ASRock ION 330 to play with, in itself this PC has nothing particular. As usual, I begin by using System Rescue CD to check bad blocks on the hard drive, before working with it (badblocks -v -s -w -c 4096 /dev/sda). Once done, I will check the memory (memtest). This is my standard procedure when receiving a new PC.

But I don't have my System Rescue CD at hand. No problem, I download it and and write it on an old CD-RW, bought 5 years ago but still sealed into his blister. For your information, this is a Memorex CD-RW 700MB 16-24x. I blank it (wodim blank=all). I write it at speed 16x, because I don't know how to lower the speed. After writing the CD, I re-read it to check everything is fine (readom dev=/dev/cdrw1 f=test.iso).

Ooops:

$ readom dev=/dev/cdrw1 f=test.iso
[...] 
Errno: 5 (Input/output error), read_g1 scsi sendcmd: no error
CDB:  28 00 00 01 3C 00 00 00 40 00
status: 0x2 (CHECK CONDITION)
Sense Bytes: 70 00 03 00 00 00 00 0A 00 00 00 00 11 05 00 00
Sense Key: 0x3 Medium Error, Segment 0
Sense Code: 0x11 Qual 0x05 (l-ec uncorrectable error) Fru 0x0
Sense flags: Blk 0 (not valid) 
cmd finished after 6.614s timeout 40s
readom: Input/output error. Cannot read source disk
readom: Retrying from sector 80896.
.............
readom: Error on sector 80908 corrected after 9 tries. Total of 0 errors.
..............---

OK, I re-blank it (wodim blank=fast) and rewrite it. The CD seems to be readable this time. But when I try to boot with it my new ASRock, it fails.

Enough is enough, I get an heavy used Verbatim 8cm CD-RW. Just to show you how old he is: the first ISO written on it was "Debian Sarge". And the magic happens: it works on the first try!!!!

You will probably think that I work for Verbatim. This is not true but I must admit that first-price CD-RW/CD-R seems to be very low quality and don't hold data in time. This is probably related to the chemical compounds of the CD. The failing Memorex CD-RW will end up in a trash and I will continue to use my 5 years old Verbatim CD-RW.

Dear reader, in order to find a way not to dump this failing CD-RW, do you know a way to make it to work?

What you should now about the failing CD-RW:

• there are no scratches on the surface
• the CD-RW has never been used before this morning
• I keep it out of the direct sunlight
• it was stored next to the Verbatim CD-RW

My wife is pregnant and we are expecting our second baby's arrival in about a month. Last time, she came back from her preparation lessons with a adverstisement "baby shower" gift pack.

One of them catch my attention:

English translation: be prepared to offer him the best...

English translation: because Debian guarantee the quality and offering the quality is a proof of love

OK, the swirl is the other way and I made a 5 minutes GIMP modification to cut-and-paste the Debian logo. But the message is here!

Remember me:

For those interested, the real thing is a soap called NUK(R).

... or as some fellow DD said, one year ago: "j'ai 1000 ans".

So what happened during last year, the DVCS way:

$> cd "Sylvain Le Gall"

$> darcs whatsnew

hunk life.txt 0
+ Yvain, my son, talks and walks, this is quite amazing to see your
+ child growing up.  My wife is pregnant, release expected in July
+ ;-)
+ 
+ I traveled quite a lot this year: Spain, Tunisia, Belgium,
+ Roscoff and Huelgoat (Brittany) 

hunk debian.txt 0
+ I went to Debconf 9 and attended Debcamp before. I met a lot of
+ very interesting people there. I spent the Debcamp time to
+ implement Zack proposal to compute OCaml package dependencies
+ using dh-ocaml. It WORKS ;-) It was a pleasure to work with Mehdi
+ Dogguy and Stephane Glondu on this subject.
+ 
+ The dh-ocaml work allowed me to be the co-author, with Stephane
+ and Mehdi, of Zack's article on the subject. 
+ My first publication!
+ 
+ I went to FOSDEM 2010 and helped attend the Debian booth. Quite a
+ nice experience I was there with my good old Open Brick NG (need
+ to blog about it one day).
+ 
+ Found some time to enable the LLVM OCaml bindings in LLVM Debian
+ package.
+ 
+ Voted for the first  OCaml-aware DPL! (NB: this was not the
+ reason for my vote)

hunk work.txt 0
+ I worked on extending a tool that I delivered before. OCaml is
+ quite a good language when it comes to do maintenance. You can
+ extend something and have dozens of compiler errors where your
+ changes propagate. Once every error has been corrected, your
+ changes are OK. Of course, this requires to use OCaml types,
+ pattern matching and a bit of unit testing.
+ 
+ I was delighted when Jun Furuse ask me if he can organize 
+ OCaml Meeting 2010 in Japan. The first OCaml Meeting spin-off. 
+ 
+ OASIS has been growing quite a lot since the year before and got
+ enough importance. Thus I decided to give a talk about it at CAML
+ Consortium in December and then at OCaml Meeting 2010. 
+ 
+ I decided to have some English training. I was missing some oral
+ fluency.
+ 
+ I organized the OCaml Meeting 2010 in April. It was quite a lot
+ of work, especially to prepare my own presentation, but the
+ meeting was a success and probably my best birthday gift with the
+ OCaml Hacking Day organized just after.

$> darcs record -a
What is the patch name? "32 year old"

Unfortunately, this release is late. My birthday was on April 16th, but it was OCaml Meeting this day, which was my top level priority at this time.

See you next year!

I hope some people from the OCaml community will enjoy this changelog, extracted from llvm 2.6-7, which has just been uploaded:

  [ Arthur Loiret ]
   
  [...]

  [ Sylvain Le Gall ]
  * Build a libllvm-ocaml-dev package, which contains the OCaml binding:
    Closes: #568556.
    - debian/debhelper.in/libllvm-ocaml-dev.{dirs,doc-base,install,META}: Add.
    - debian/control.in/source: Build-Depends on ocaml-nox (>= 3.11.2),
      ocaml-best-compilers | ocaml-nox, dh-ocaml (>= 0.9.1).
    - debian/packages.d/llvm.mk:
      + (llvm_packages): Add libllvm-ocaml-dev.
      + (libllvm-ocaml-dev_extra_binary): Define, install META file.
    - debian/rules.d/binary.mk: Add dh_installdirs and dh_ocaml.
    - debian/rules.d/vars.mk:
      + include /usr/share/ocaml/ocamlvars.mk.
      + Configure with --with-ocaml-libdir=$(OCAML_STDLIB_DIR)/llvm.
  * debian/rules.d/build.mk: Fix symlinks pointing to the $DESTDIR.

In other words: LLVM is now built with its OCaml bindings and a META file for findlib. It will take some days before reaching every architectures, but hopefully it will be in Squeeze (next Debian stable release).

Thanks to Arthur Loiret for the quick upload.

Last year, I was not able to attend FOSDEM due to last minute problems. However, this year I will be there and even attend the Debian for some periods ([http://wiki.debian.org/DebianEvents/FOSDEM/2010]).

I will bring my Openbrick NG with a standard Debian Lenny and probably a Babelbox installation. The Openbrick is a VIA C3 fanless computer. It is not very exotic but it is quite interesting to see this kind of hardware. For years, I have tried to build/use fanless computer. This is not a very popular topic but it introduces problems of heat and noise at a higher level.

I am still setting up the Babelbox, which should have been a Debian Lenny RC1. I will try to upgrade it to Debian Lenny 5.0.3.

See you at FOSDEM 2010.

During one of my project I need to interact with Java cryptographic extension. Some data has been encrypted using PBEWithMD5AndDES. I need to access it from OCaml.

I take a look at available cryptographic extension in the Debian project for OCaml: cryptgps and cryptokit. I choose cryptokit, because its author is well known: Xavier Leroy.

This article was my starting point. Of course, I keep in mind that the reference is there and that there is a good article covering it.

Here is the result in OCaml:

 let decrypt passphrase salt ?(iterationCount=41) str =
   let key, iv =
     let rec hash_aux iter str =
       if iter > 0 then
         (* Rehash string *)
         hash_aux
           (iter - 1)
           (hash_string
              (Hash.md5 ())
              str)
       else
         (* Key = first 8 bytes of the MD5 hash *)
         String.sub str 0 8,
         (* IV = last 8 bytes of the MD5 hash *)
         String.sub str 8 8
     in
       (* Hash n times combination of passphrase and salt,
           return key and iv 
         *)
       hash_aux
         iterationCount
         (passphrase ^ salt)
   in
     transform_string
        (Cipher.des
           ~pad:Padding.length
           ~iv:iv
           key
           Cipher.Decrypt)
       str

The only missing information was the pad algorithm to use (Padding.length). For this piece of information, I need to browse the RSA documentation and test a little bit.

Rewriting PBEWithMD5andDES is quite straightforward with cryptokit and OCaml. It takes 25 lines with C# and OCaml (only counting LoC, no comment, no empty constructor or declaration in C#). I was thinking that this task will require 2 or 3 days, but it has been done in 4 hours...

Many thanks to cryptokit ;-)

I am back from Debconf 9 in Cáceres. As usual it was a great time to meet other Debian contributor and to exchange knowledge.

This year we come with several new and old pieces of OCaml code that has attracted unexpected people to OCaml:

• the debian release team ask us for some demonstration of the language in order to see if the transition monitor is usable for general transition tracking
• Joachim Breitner is now using edos-debcheck to enhance test before starting build (should save time if some package are uninstallable)

Git repository of dh-ocaml

During Debcamp we, the OCaml Debian task force, have done a big work on dh_ocaml and related tools:

• dh_ocaml can now computes Depends/Provides and .md5sums file for every OCaml package
• dh_ocamldoc can produce documentation and is a replacement for CDBS ocamldoc targets
• dh_ocamlinit do replacement in debian/*.in files just as ocamlinit.mk before
• lintian tests (bug submitted to lintian for integration)
• policy review (still in progress)

The dh_ocaml is working as follow:

• it makes the distinction between 3 kinds of packages: dev, runtime, and (standard) binary
  • dev and runtime package are always associated since runtime just provide non-development objects of a dev package
  • binary package that are not dev neither runtime package
• dev and runtime packages are scanned for objects containing assumptions (*.cmx, *.cmi, *.cmo...)
• all packages are scanned for bytecode executables that also contain assumptions
• assumption are extracted and sorted into 2 categories: defined and imported
• defined assumptions are used to compute interfaces provided by the package
• imported assumptions are used to compute interfaces from which Depends the package
• defined assumption is used to generate .md5sums file that will be distributed in the dev package and will be used to compute Depends for other packages

Provides and Depends use virtual packages and checksums of the defined assumptions. E.g. it use libzip-ocaml-dev-12z3a to stat dependency to libzip-ocaml-dev, the 12z3a is the base 36 checksum of the 5 first digits of the MD5 of defined interfaces. This way, if some interfaces change in camlzip, checksums will also change and will make package that depends on libzip-ocaml-dev-12z3a not installable. It should prevent user to install packages that don't make good assumptions about libzip-ocaml-dev.

Package "ocaml" (resp. "ocaml-nox") is a dev package, its runtime is "ocaml-base" (resp "ocaml-base-nox"). In the case of this OCaml base packages, we use the upstream version rather than the checksum in the Provides/Depends. So provided packages are "ocaml-3.11.1" (runtime "ocaml-base-3.11.1"), "ocaml-nox-3.11.1" (runtime "ocaml-base-nox-3.11.1"). In fact, using this scheme we just fallback to the earlier way of making Depends in OCaml Debian packages, except that now it is automatically computed and generalized

Depends are computed using the type of packages: runtime and binary package will Depends on other runtime packages, dev packages will Depends on other dev packages. E.g. headache is a bytecode executable, this package is classified as a binary package. dh_ocaml automatically make it Depends on "ocaml-base-nox-3.11.1" because it contains bytecode objects that used it. Another example, libzip-ocaml-dev is a dev package and dh_ocaml make it Depends on "ocaml-nox-3.11.1".

We have checked on a various packages this Depends/Provides field completion and it seems to work quite well. We have found unexpected dependencies (ocaml-findlib depending on "ocaml-3.11.1" because it contains a wizard using Tk). We have also found unexpected defined interface (ocaml-fileutils defines module Unix due to a "-linkall" flag when building the .cma).

What remains to do:

• dh_ocaml needs to make the difference between arch-independent and dependent provides
• dh_ocaml needs to do binNMU safe version between internal packages (i.e. ones being built)
• some more tools: dh_ocamlmeta to install META files (with good version), dh_ocamldirs to create standard directories, dh_ocamlinstall to move objects in dev and runtime package from debian/tmp build directory.

All this new tools should help to track automatically dependencies between packages and ease the work of OCaml debian task force. Debcamp was a great time for a brainstorm on Debian OCaml package management. The overall quality of OCaml in Debian should be increased.

This year I will spend 2 weeks at Debcamp and Debconf9. With some other members of the Debian OCaml Task Force, we will try to improve dh-ocaml.

In particular, we will try to implement a proposal by Stefano Zacchiroli for OCaml link-time compatibility and Debian dependencies. This proposal, approximated ABI in the paper, should allow to reflect through Debian dependencies which OCaml libraries are compatible. For now relationship between OCaml packages are quite naive, since we only use version operation like: >=, >>, <=... These operations are not precise enough with regard to OCaml assumption about modules. This proposal will improved dependencies by using virtual package that will make a summary of OCaml assumption.

If time and workforce is enough, maybe it should also be a good time to refesh our policy a little bit and take a look at implementing lintian check for it. But this only "bonus objectives" for this year Debcamp/Debconf.

Last time Debconf7 has been a great place to exchange idea, to delve into debian subjects and to discover totally amazing thing. The rest of the year, I am always busy with real life and urgent debian work. Debcamp and Debconf will be a great time to stay for a long period on the same subject, trying to find the best solution.

Next week will be quiet busy. I will travel a lot. Of course, I am talking about OCaml Meeting and FOSDEM.

The OCaml Meeting subscription is over and we reach a quiet good number of attendees: 45. This is the approximate number of people who come last year. There is now some details to finish (dinner for the previous day, schedule for talks...) but enough time left. The only thing I hope for the meeting is that it generates the same effect as last year: another impulse in the OCaml community. Many people are working continuously on OCaml but last year event has generated a lot of post-event discussion and many people started different project. Some of these projects are still alive, some are not as vivid as they deserve.

I enjoy being at FOSDEM last year, but I only stay 1 day and it was not enough. I will be able to stay during the 2 days and attend more Debian conference than last year.

I will visit Brussels and Ghent just after. Belgium is a really nice country and it has been a long time I haven't take a break.

The subscription will end on January the 25th, for people willing to come to this unique OCaml event, this is your last chance !!!!

We have 40 participants for now. I am quite happy with this number because it is almost the same as last year -- except that we do it in Grenoble rather than Paris. Anyway, I see that a lot of people who were there last year, will come back to this year events. There is also a few more people, in particular more from CAML consortium members.

I have stepped down from the first talk about OCamlCore.org in favor of Stefano Zacchiroli. Zack has really helped me on this project since the begin (with Pietro Abate and Romain Beauxis). I am very happy that he can do the talk.

I will do myself, a talk about my last project for Talend. This was a great project targeting to be as fast as a standard C application but in OCaml. The point is that OCaml helped me to do quickly a working prototype and then optimize it for speed. All in all, after 6 months I was able to deliver Talend with a working and fast application. This kind of release cycle is harder to achieve in other programming language. OCaml compile time verification helped me a lot.

I hope that in the last remaining 2 days, some more people will subscribe.

Wiki Subscription

This morning, Alan sends me the link to the subscription form. We are now ready to accept participants for OCaml Meeting 2009.

Last week, I was at the annual CAML consortium meeting. We talked about OCaml Meeting with M. Leroy and other participants. Most of us, think this meeting is a real great way to make OCaml user meet. This can be the start of great projects, just because people realized that a real OCaml community exists. The INRIA seems really enthusiast about the meeting, maybe INRIA team will be directly involved in the organization in 2010.

I hope 2009 meeting will be as good as the former. At least this year, we are doing things quite in advance and the whole organization will be a little bit more "professional".

OCaml Meeting webpage

Unfortunately I forget to post the last year participant pictures. Here is the official picture.

Thanks to R.W.M Jones for reminding me.

Since September 2008, Alan Schmitt and I have decided to organize a new OCaml Meeting following the JFLA. This will be the second time of such an event. Last year we were 50 people -- after a very short period for organizing it (less than a month). This year we decided to prepare things earlier.

As a matter of fact, the last year events has at least generated a peak in OCaml mailing list (602 message in January, 487 in february...) It also has given birth to several things like planet and forge. It has also help to start things like batteries. I hope this year events will help people to meet and exchange great ideas for OCaml promotion.

Waiting to see a lot of "OCaml"ers at OCaml Meeting 2009.

Having already done some Debian packages of OCaml libraries, I wish to share my experience on what are the most common problems. The intent is to help people releasing OCaml libraries to have a good interaction with (Debian) packager. I think most of the tips should also apply to other distributions as well. I consider the use of findlib mandatory for this task as described in the Debian OCaml packaging policy.

Problems listed here are common, and I have encountered them for my libraries... In fact, most of the time when I finished building a personnal OCaml library I package it for my own and to test that everything is fine. Being a packager and an upstream author is really a different job.

"Upstream author" is used for people developing and releasing OCaml libraries and "packager" for people doing packages for a distribution.

Here is the list of the most common problems and their solutions:

• Missing the "static link clause" in COPYING/LICENSE

Libraries are linked statically in OCaml. It is an advantage and a problem. But upstream author should remember that the LGPL license doesn't support static link directly. If you link an executable with the library statically, the executable is contaminated by the license. This error is probably the most problematic, because it requires to go through the whole project fixing license header for explaining that there is an exception to the LGPL.

There is no way for a packager to fix this problem. Packager are not allowed to change the license. Library having this problem can be still packaged but complicate the legal aspect of building application using this library.

Good example of this can be found in OCaml LICENSE itself :

As a special exception to the GNU Library General Public License, you
may link, statically or dynamically, a "work that uses the Library"
with a publicly distributed version of the Library to produce an
executable file containing portions of the Library, and distribute
that executable file under terms of your choice, without any of the
additional requirements listed in clause 6 of the GNU Library General
Public License.  By "a publicly distributed version of the Library",
we mean either the unmodified Library as distributed by INRIA, or a
modified version of the Library that is distributed under the
conditions defined in clause 2 of the GNU Library General Public
License.  This exception does not however invalidate any other reasons
why the executable file might be covered by the GNU Library General
Public License.

Followed by the text of the LGPL.

Each file header looks like:

This file is distributed under the terms of the GNU Library General
Public License, with the special exception on linking described in 
file ../LICENSE.

• Forget that some architecture doesn't have an ocamlopt compiler

By far this is the most common error I have seen. Upstream author consider that you are able to build a native and byte version of their library. Unfortunately this is not the case for most architectures. This error is pretty simple to fix by separating byte and opt targets in Makefile. The ultimate test is to rename /usr/bin/ocamlopt and /usr/bin/ocamlopt.opt to something different and try to rebuild the library from scratch.

• Missing or incomplete META file

META file is the base of everything in findlib. This is very important to provide good META file because it helps other people to easily build thing over OCaml library. Each library should provide its own META file. Until now, Debian packager was used to write this missing META file and contribute it to upstream author. I think that now findlib has become a standard in OCaml developement, so upstream should take into consideration this file.

A good test for META file is to install it and:

~$ ocaml
        Objective Caml version 3.10.1

# #use "topfind";;
- : unit = ()
Findlib has been successfully loaded. Additional directives:
  #require "package";;      to load a package
  #list;;                   to list the available packages
  #camlp4o;;                to load camlp4 (standard syntax)
  #camlp4r;;                to load camlp4 (revised syntax)
  #predicates "p,q,...";;   to set these predicates
  Topfind.reset();;         to force that packages will be reloaded
  #thread;;                 to enable threads

- : unit = ()
# #require "ZZZ";;

If you are able to load your library in ocaml toplevel, this is a good point.

• Doesn't use ocamlfind to install library

ocamlfind allow to install library. Using it is the best way to install your ocaml library. Moreovoer, you should use the most simple way to install.

For pure OCaml library this is straightforward and allow Debian packager to override destination directory by using environment variable OCAMLFIND_DESTDIR.

install:
      ocamlfind install toto META toto.mli toto.cmi toto.cmxa toto.cma toto.cmx

For non-pure library, you could add a OCAMLFINDFLAGS variable to allow Debian packager add "-ldconf ignore":

install:
      ocamlfind install $(OCAMLFINDFLAGS) toto META toto.mli toto.cmi toto.cmxa toto.cma toto.cmx

• Doesn't distribute .mli files

".mli" files is the human-readable interface to the library. Even if most OCaml programmers use the ocamldoc HTML API of the library, it is best to distribute these files. In particular, Debian packager use it to automatically generate up-to-date HTML API documentation directly.

• Doesn't distribute .cmx files

".cmx" files contain more informations than ".cmxa". Distributing these files allow application based on the library to be more optimized.

• Forget about other OS files

This one comes from my "Windows experiment". Most of the time upstream author doesn't forget about "*.a" but doesn't add "*.lib" (yes is this windows counterpart of ".a" file).

• Use bad wildcard for file to distribute

This one is not very OCaml specific. In Makefile and sh you have the same wildcard "*.mli". But if the wildcard is not expanded into sh, it just stay as is. It will make the executable called looked for a file "*.mli" literally. Mixing Makefile wildcard and sh is the best option here:

install:
     ocamlfind install $(OCAMLFINDFLAGS) toto META $(wildcard *.mli *.cmi toto.cmxa toto.cma *.cmx)

This way you test and expand to only existing files.

• Use a custom build system

This is more or less a problem. In fact, if everything is fine, there is no problem and you should just forget about this point. If the custom build system is complex and if there is an error in it: it becomes a big problem.

I must confess that I have myself done this kind of mistake a lot of time. But looking back, I think that most of the time I should have used directly OCamlMakefile. I think relying on ocamlfind for library installation, simple Makefile, OCamlMakefile, OMake or ocamlbuild, is just enough.

• Missing BTS

This point is a problem for upstream author and communication with packagers (from Debian and other distributions). A bug tracking system is a way to publish, comment, work on and find solution of bugs. If you have a single problem in a library having the bug public helps people to show you what is wrong.

Of course, Debian (and Fedora, FreeBSD...) has its own bug tracking system and upstream author can use it directly. This is less work for upstream but tends to make the bug "distribution specific". Having an upstream author BTS helps to track the bug across distributions.

If upstream author doesn't like maintaining their own BTS, he/she should consider hosting part of the project on a forge like OCamlCore.org. Even if they don't host the source code, this is still a good idea.

That's all. If you have other common problems, feel free to comment, i will add it to the list.