Blog of Sylvain Le Gall

To content | To menu | To search

Thursday, April 4 2013

Sekred a password helper for puppet.

Puppet is a nice tool but it has a significant problem with passwords:

  • it is recommended to store puppet manifests (*.pp) and related files in a VCS (i.e. git)
  • it is not recommended to store password in a VCS

This lead to complex situation and various workaround that more or less work:

  • serve password in a separate file/DB or do an extlookup on the master (pre-set passwords)
  • store password on the server and get them through a generate function (random password but on the master)

Most of these workarounds are complex, don't allow you to share the password you have set easily and most of the time are stored in another place than the target node.

So I have decided to create my own solution: sekred (LGPL-2.1).

The idea of sekred is to generate the password on the target node and made it available to the user that needs it. Then the user just have to ssh into the host and get the password.

Pro:

  • the password is generated and stored on the node
  • no VCS commit of your password
  • no DB storage of your password beside the local filesystem of the host
  • no need to use a common pre-set password for all you host, the password is randomly generated for only one host
  • to steal the password you need to crack the host first but if you have root access on the host, accessing a random generated password is pointless

Cons:

  • the password is stored in clear text
  • the password is only protected by the filesystem ACL

Let see some concrete examples.

Setting mysql root password

This is a very simple problem. When you first install mysql on Debian Squeeze, the root password is not set. That's bad. Let set it using sekred and puppet.

node "mysqlserver" {

  package {
    ["mysql-server",
     "mysql-client",
     "sekred"]:
      ensure => installed;
  }

  service {
    "mysqld":
      name => "mysql",
      ensure => running,
      hasrestart => true,
      hasstatus => true;
  }

  exec {
    "mysql-set-root-password":
      command => "mysqladmin -u root password $(sekred get root@mysql)",
      onlyif => "mysql -u root",  # Trigger only if password-less root account.
      require => [Service["mysqld"], Package["mysql-client", "sekred"]];
  }
}

And to get the root password for mysql, just login into the node "mysqlserver":

$> sekred get root@mysql
Cie8ieza

Setting password for SSH-only user

This example is quite typical of the broken fully automated scenario with passwords: - you setup a remote host only accessible through SSH - you create a user and set its SSH public key to authorize access - your user cannot access its account because SSH prevent password-less account login!

In other word, you need to login into the node, set a password for the user and mail him back.... That defeats a little bit the "automation" provided by puppet.

Here is what I do with sekred:

define user::template () {
  user {
    $name:
      ensure => present,
      membership => minimum,
      shell => "/bin/bash",
      ....
  }
  include "ssh_keys::$name"

  # Check password less account and set one, if required.
  $user_passwd="$(sekred get --uid $name $name@login)"
  exec {
    "user-set-default-password-$name":
      command => "echo $name:$user_passwd | chpasswd",
      onlyif => "test \"$(getent shadow $name | cut -f2 -d:)\" = \"!\"",
      require => [User[$name], Package["sekred"]];
  }
}

So the command "test \"$(getent shadow $name | cut -f2 -d:)\" = \"!\"" test for a password-less account. If this is the case, it creates a password using sekred get --uid $name $name@login and set it through chpasswd.

Note that $user_passwd use a shell expansion that will be evaluated when running the command only, on the host. The --uid flag of sekred assign the ownership of the password to the given user id.

So now the user (foo) can login into the node and retrieve its password using sekred get foo@login.

Try it!

Sekred was a very short project but I am pretty happy with it. It solves a long standing problem and helps to cover an extra mile of automation when setting up new nodes.

The homepage is here and you can download it here. Feel free to send patches, bugs and feature requests (here, login required).

Saturday, March 2 2013

Always test your HD first

I just received a WD 500GB Blue, to replace the hard drive of my wife's computer. First thing after unpacking: start to test. And I was right to do it... there are bad blocks on it.

WD 500GB DOA

The story: I tend to do proactive replacement of hard drive, to avoid losing too much data or be in hurry when the current one fails. There are two ways to monitor a hard drive:

Logcheck is pretty straightforward to install. It scans your log every two hours and send you a report of what is happening. It is not a very precise tool and you have to tune it a little bit to just send you what is relevant (install extra rules to ignore what you know is not of interest). Whenever you start to see log entry like that:

Jan 24 18:16:08 foo kernel: [ 1965.343980] ata5.00: exception Emask 0x50 SAct 0x39 SErr 0x800 action 0x6 frozen
Jan 24 18:16:08 foo kernel: [ 1965.343991] ata5.00: irq_stat 0x08000000, interface fatal error
Jan 24 18:16:08 foo kernel: [ 1965.344001] ata5: SError: { HostInt }
Jan 24 18:16:08 foo kernel: [ 1965.344036] ata5.00: failed command: READ FPDMA QUEUED
Jan 24 18:16:08 foo kernel: [ 1965.344055] ata5.00: cmd 60/08:00:18:31:44/00:00:1a:00:00/40 tag 0 ncq 4096 in
Jan 24 18:16:08 foo kernel: [ 1965.344059]          res 40/00:2c:e6:c2:fd/00:00:26:00:00/40 Emask 0x50 (ATA bus error)
Jan 24 18:16:08 foo kernel: [ 1965.344071] ata5.00: status: { DRDY }
Jan 24 18:16:08 foo kernel: [ 1965.344081] ata5.00: failed command: WRITE FPDMA QUEUED

It is a good time to think about changing your hardrive -- but it is maybe too late.

Smartmontools (aka smartd) is dedicated tool to monitor hard drives and do a good job. I think it is not installed by default in Debian, but it should be. It scans your hard drive for SMART capabilities and monitor the health of the HD using internal tools. In the case of bad blocks, you will start to see entry like that:

Feb 17 14:05:17 bar smartd[1268]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Feb 17 14:35:17 bar smartd[1268]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors

Obviously when you see this log, it is also a good time to change your hard drive.

In the case of my wife's HD, I just got data from logcheck. It means that the error is not that important (transient failure, something is wrong but the HD can cope with it). But I still decided to get a new one for my wife.

Whenever, I receive a new drive, the first thing I do is to check it for errors. You can do that using the program badblocks in write mode. It takes ages to test (count up to 1 day for 1TB on USB), but at the end you know that you have a good candidate -- where it is worth install your data.

You just have to follow this procedure

  1. dmesg | grep sd
  2. try to find what drive is the one you want to test in the output of 1.
  3. cfdisk /dev/sdX, sdX being the drive you want to test
  4. check that what you see in cfdisk is what you expect to test: right name and capacity for the drive
  5. sudo badblocks -wvs /dev/sdX
  6. run sudo tail -f /var/log/syslog in parallel just in case
  7. wait

If some errors appears in /var/log/syslog, you know something bad is happening. Whenever you have a single failing block, don't think it is ok. It is NOT ok for a HD to start its life with failing blocks. In this case, repack the drive and send it for replacement ASAP.

In the case of my hard drive: smartd mail:

The following warning/error was logged by the smartd daemon:

Device: /dev/sda [SAT], 15 Currently unreadable (pending) sectors

syslog entries:

Mar  2 20:52:57 foo kernel: [ 8317.419715] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
Mar  2 20:52:57 foo kernel: [ 8317.419724] ata4.00: irq_stat 0x40000008
Mar  2 20:52:57 foo kernel: [ 8317.419732] ata4.00: failed command: READ FPDMA QUEUED
Mar  2 20:52:57 foo kernel: [ 8317.419747] ata4.00: cmd 60/80:00:00:af:08/00:00:28:00:00/40 tag 0 ncq 65536 in
Mar  2 20:52:57 foo kernel: [ 8317.419750]          res 41/40:00:65:af:08/00:00:28:00:00/40 Emask 0x409 (media error) <F>
Mar  2 20:52:57 foo kernel: [ 8317.419758] ata4.00: status: { DRDY ERR }
Mar  2 20:52:57 foo kernel: [ 8317.419764] ata4.00: error: { UNC }
Mar  2 20:52:57 foo kernel: [ 8317.423959] ata4.00: configured for UDMA/133

I am not blaming any particular brand (like Western Digital), all computer parts I have ever bought had to follow the same procedure and it is a known fact that computer parts as a non-zero percentage of chance to be DOA (dead on arrival) or after a few weeks. But as a consumer you should be aware of that and take action to avoid spending 10h configuring your computer to see it failing after a week... The waste of time to test is a win on the long term.

Wednesday, February 27 2013

planet.ocaml.org spring cleaning

Hi planet.ocaml.org.

Just a quick post to thanks Marek Kubica for his help on the planet.ocaml.org spring cleaning.

Here are the feeds that have been removed:

  • Red Lizard Software

http://redlizards.com/blog/feed/?tag=ocaml

  • Alp Mestan

http://blog.mestan.fr/feed/?cat=16

  • Arlen Cuss

http://www.sairyx.org/tag/ocaml/feed/

  • Daniel Patterson

http://blog.dbpatterson.com/rss

  • Victor Nicollet, cannot find any feeds on the new blog

http://www.nicollet.net/toroidal/ocaml/feed/

  • OCaml Hackers

http://ocamlhackers.ning.com/profiles/blog/feed?tag=ocaml&xn_auth=no

  • Mauricio Fernandez, offline

http://eigenclass.org/R2/feeds/rss2/all

  • Christopher Conway, no OCaml-related posts since 2008

http://procrastiblog.com/category/ocaml/feed

  • Liquidsoap, cannot find the HTML blog on the website

http://savonet.sourceforge.net/liquidsoap.rss

Here is the feed that have been added:

  • Marc Simpson

http://newblog.0branch.com/rss.xml

Here are the feeds that have been updated:

  • Jane Street Capital now points to

https://ocaml.janestreet.com/?q=rss.xml

  • Jamie Brandon now points to

http://scattered-thoughts.net/atom.xml

  • Mihamina Rakotomandimby now points to

http://www.rktmb.org/feed/tag/ocaml/atom

  • Dario Teixeira now points to

http://nleyten.com/feed/tag/ocaml/atom

  • Erik de Castro Lopo now points to but FP-Sydney has been removed

http://www.mega-nerd.com/erikd/Blog/index.rss20

  • Y-Node now points to

http://y-node.com/blog/feeds/latest/

If you want that we had back your blog, please follow the howto add your feed to planet. We didn't have removed feed on purpose, this was just a way to get rid of a lot of 404,

And don't forget, planet.ocamlcore.org is now served by planet.ocaml.org! Update your feed reader.

Saturday, February 23 2013

OUnit 2.0 progress

I have recently started to work on OUnit 2.0. The point of this new version will be to adapt OUnit to improve speed and compatibility with third party system:

  • better configuration setup (through the use of environment variable, command line options and configuration files)
  • systematic logging (verbose always on), but output log in a file
  • allow vim quickfix to jump in the log file where the error has happened
  • output HTML report
  • output JUnit report
  • run tests in parallel
  • automatic selection of tests (choose the one that failed in the last run, before re-running the one that was ok)

Only the 3 last point needs to be completed.

Here is a screenshot of Jenkins reading JUnit output of an OUnit test: Jenkins, OUnit test results

So if you have a long standing issue with OUnit, this is the time to submit a bug. OUnit BTS

Friday, October 19 2012

Configuration management: Puppet is worth it.

Replying to an old blog post of Martin F. Krafft: Configuration management, I want to give my point of view.

The problems listed by madduck are quite common with Puppet, but I think Puppet is still worth, mostly because you can solve all these problems.

Let give you my opinion on the list:

  • Non-Unix approach to everything (own transport, self-made PKI, non-intuitive configuration language, a faint attempt at versioning (bitbucket), and much much more…)

True. I think the approach of puppet is not really UNIXish. It is probably on purpose. The biggest issue is probably the PKI. It breaks frequently for unknown reason. The "non intuitive configuration language" is probably a matter of taste. I think the language is not very well designed and strange, but I can cope with that. The attempt to versioning -- if I understand correcly what it means -- refers to the fact that when Puppet replace a file it moves the old file to a bucket. This is not a good thing, but you can say "backup => '.puppet-bak'" and you get almost the same behavior as ".dpkg-old".

  • Ruby

False debate. We can discuss for hours on Ruby, PHP, Java or whatever pet language people has invented. I am not a fan of Ruby but it is still nice as a general purpose language. To my mind, Ruby is still better to write daemon than bash.

  • Abysmal slowness

False debate.

   info: Caching catalog for centi.....
   info: Applying configuration version '1350597216'
   notice: Finished catalog run in 3.08 seconds

The config of this node is not complex, but 3s is not that bad for something that runs every 30min. If you need sub-second speed for this kind of thing, maybe you are not looking for this kind of tool. Does 144s of server time per day is a big deal ?

With a lot more complex setup, I can reach 30s for a run, although this is the point where I manage a lot of thing with it.

  • Lack of basic functionality (e.g. replace a line of text)

False and True. Augeas allows you to replace a single value (even more precise than a line). Just have a look at the augeas type. This is pretty nice and allow to do thing like replacing "Defaults env_reset" by "Defaults env_reset, !tty_tickets" in 4 lines of code. So this i not precisely "a single line of text", but there is other way to do it.

  • Host management and configuration programming intertwined, lack of a high-level approach to defining functionality

False. Well if you organize your code with manifests/site.pp and manifests/classes/*.pp, it seems like there is a separation between the two. Next you can try inheritance and define to create specific high-level features.

  • Horrific error messages

False-ish. Hey at least there are error message ;-) Now, most of the error that are related to the programming language are useless (at least as cryptic as a C++ error message). But as usual with error message in programming language

  • Catastrophic upgrade paths

True. Multi versions installation is horrible and you have to fix a lot of stuff to manage a sane overall configuration.

  • Lack of IPv6 support

Not sure to understand this point, I use puppet over IPv6...

To whoever is considering using puppet, this is worth a try. It is a nice system that really helps to maintain a decent configuration across nodes.

Friday, October 5 2012

Book review: Puppet 2.7 Cookbook

I recently read this book and it was enlightening -- that is the least I can say. It is organized into a set of recipes addressing the most common problems you have with puppet. I think the whole point is about being practical and it gives you various ways to achieve the same goal, depending if you want to do a quick and dirty hack or pick a long term solution. The whole book aims at being practical and it achieves this goal pretty well. However, it is not for beginners. Puppet itself is not for beginner and you should already have written and run into problem to really take advantage of this book.

I read the book in 5 days (only while commuting), and at each page I thought : "OMG, they do that like this" or "of course, you should use git + that" and wait impatiently coming back home to test all this stuff. The book is really about technical details and how to organize yourself to write nice Puppet classes. I have been an "irregular" user of Puppet since 4 years. I was mainly running it to distribute files on all my computers. Since 3 weeks, I am applying the recipes found in this book and it is just like discovering that you own something worth a million euros.

Let look at some example:

  • Setting a value in a configuration file:

Before: Replace the whole file.

Now: Use Augeas to just change the value and don't touch to the rest of the file. The book makes me discover Augeas and it rocks !!!

  • VCS repository and deployment:

Before:_ Loosy darcs repository inside /etc/ and edit directly the master.

Now: Shiny git repository in my home directory, using a Makefile to validate Puppet syntax, simulate applying it and effectively apply it to the current node before deploying it through git push.

  • Installing exim4:

Before: Nothing because I didn't think it was possible.

Now: Generate a configuration file using a template, install it, run the exim4 update script and restart the service.

  • Installing sshd:

Before: Copy a file and restart it.

Now: Use Augeas and a template with a loop to change what is needed for each node depending on which user I want to allow to connect on this node. Use different SSH port for every vserver that share the same IP and distribute an /etc/ssh_config to propagate the default ports set for all the vservers... (just being able to do that is worth the price of the book).

  • Have a problem and try to solve it:

Before: Complain, whine and give up.

Now: Go to Puppet Forge, take inspiration for similar module, re-read part of the book, find an idea and apply it.

And so forth and so on. This book made me love puppets again. I warmly recommend it to you.

Wednesday, July 18 2012

Coding by example, migrating ocsoap to OASIS and ocamlbuild

In my effort to automate most of the stuff needed to release, I am working on a SOAP to OCaml converter. Richard W.M. Jones has kindly allowed me to take-over its project "ocsoap". The project is now located here and I am working on making it compatible with FusionForge SOAP.

My first step was to make it compatible with OASIS and ocamlbuild. The project is Makefile based and needs some extra care to make it ocamlbuild based.

The oasis-fication itself was a piece of cake. Just describe the dependencies of the project and make some extra choice like compiling the camlp5 extension to a .cma rather than a .cmo. You can see the _oasis and myocamlbuild.ml results, compared to the initial Makefile.

The _oasis is simple:

OASISFormat:  0.3
Name:         ocsoap
Version:      0.7.1
Synopsis:     SOAP converter to OCaml code
Authors:      Richard W.M. Jones, Sylvain Le Gall
License:      LGPL-2.1 with OCaml linking exception
Plugins:      DevFiles (0.3), META (0.3), 
              StdFiles (0.3)
BuildTools:   ocamlbuild,
              cduce
BuildDepends: dynlink,
              pxp-lex-utf8,
              pxp-engine,
              netclient,
              cduce,
              extlib,
              calendar,
              pcre

Library ocsoap
  Path:       src
  Modules:    OCSoap

Library pa_ocsoapclientstubs
  Path:          src
  Modules:       Pa_ocsoapclientstubs
  BuildTools:    camlp5o
  BuildDepends:  camlp5
  FindlibParent: ocsoap
  FindlibName:   syntax
  CompiledObject: byte
  
Executable wsdl_validate
  Path:       src
  MainIs:     wsdl_validate.ml
  
Executable wsdltointf
  Path:       src
  MainIs:     wsdltointf.ml

Executable adwords_test1
  Path: examples/adwords
  BuildDepends: ocsoap
  MainIs: test1.ml
  Build$: flag(tests)
  Install: false
  BuildTools: camlp5o
  BuildDepends: ocsoap.syntax

Executable adwords_test2 
  Path: examples/adwords
  BuildDepends: ocsoap
  MainIs: test2.ml
  Build$: flag(tests)
  Install: false
  BuildTools: camlp5o
  BuildDepends: ocsoap.syntax

Executable adwords_examples 
  Path: examples/adwords
  BuildDepends: ocsoap
  MainIs: example.ml
  Build$: flag(tests)
  Install: false
  BuildTools: camlp5o
  BuildDepends: ocsoap.syntax

Document "api-ocsoap"
  Title: API reference of OCSoap
  Type: ocamlbuild (0.3)
  BuildTools+: ocamldoc
  XOCamlbuildLibraries: ocsoap
  XOCamlbuildPath:      src/

Most of it was generated using oasis quickstart, that helps you to write _oasis. This part was not tricky and mostly a translation of what the previous Makefile was explaining, in its language.

Now comes the tricky part, translating Makefile rules into ocamlbuild rules. This part is more tricky because the general principle behind Makefile and ocamlbuild are not exactly the same.

Let's start with a simple rule: compiling a CDuce .cd file into a .cdo.

Here is the Makefile rule:

%.cdo: %.cd
        $(CDUCE) --compile $<

In the oasis-fication, we have added an extra complexity, because we move the source to src, which needs extra flags.

Here is the ocamlbuild rule:

rule "cduce: %.cd -> %.cdo"
  ~prod:"%.cdo"
  ~dep:"%.cd"
  begin
    fun env build ->
      Cmd(S[cduce;
            T(tags_of_pathname (env "%.cd")++"cduce"++"compile");
            A"--compile";
            A"-I"; P(Filename.dirname (env "%.cd"));
            P (env "%.cd")])
  end
;;

It is a little bit more complex, lets explain it.

The function rule create a rule with a name and a function that execute it. The labels ~prod and ~dep are the right and left parts of %.cdo: %.cd of the Makefile. When we call env "%cd", it replaces the % by the matching part of ~prod and ~dep.

Next we return a Rule.action. The rule in this case is a command Cmd. This rules use a sequence S, which are the command line itself. The sequence is made of smaller pieces that can be a placeholder for tag content (T + what will trigger it), atom (A, typically command line option), and filename (P).

For a file src/foo.ml, the command generated will

cduce $(TAG) --compile -I src src/foo.ml

where $(TAG) will be replaced by the content of flag ["file:src/foo.ml"; "cduce"; "compile"] content. But also by the content of flag ["cduce"; "compile"] content, the flag just need to be subset. For example if you want to add --verbose to the command line, just add flag ["cduce"; "compile"] (S[A"--verbose"]);; somewhere in myocamlbuild.ml.

Next rules, we need to compile %.cdo to %.cmo. It is trickier because in this case, we want to have %.cmi eventually built before. It is not require to build it before, if there is no %.mli file.

Here is the code to do that:

let cduce_mkstubs includes =
  Quote (S([cduce;
            S(List.map
                (fun fn -> S[A"-I"; P fn])
                includes);
            A"--mlstub"]));;
(* cduce < 0.3.9:  cdo2ml -static *) 

let cduce_compile_args env =
  [
    A"-c"; A"-package"; A"cduce";
    A"-pp"; cduce_mkstubs [Filename.dirname (env "%.cmi")];
    A"-I"; P(Filename.dirname (env "%.cmi"));
    A"-impl"; P(env "%.cdo")
  ]
;;

let cduce_prepare_compile env build =
  List.iter
    (function
       | Outcome.Bad _ ->
           (* Fail but it just means that the .cmi will be generated 
            * during compilation.
            *)
           ()
       | Outcome.Good _ ->
           ())
    (build [[env "%.cmi"]])
;;

rule "cduce: %.cdo -> %.cmo"
  ~prod:"%.cmo"
  ~dep:"%.cdo"
  begin
    fun env build ->
      cduce_prepare_compile env build;
      Cmd(S(
        [ocamlfind; ocamlc;
         T(tags_of_pathname (env "%.cdo")
           ++"cduce"++"ocamlc"++"compile"++"byte")]
        @ (cduce_compile_args env)))
  end
;;

The difference with the former rules, is that we call cduce_prepare_compile which in turns call (build [[env "%.cmi"]]). The call to build will ask ocamlbuild to compile src/foo.cmi, but we don't care about the result, i.e. in case of Outcome.Bad exn we don't fail. This way we don't stop the build process that will continue and produce a .cmo and .cmi just out of the .cdo. The .cdo itself is compiled as a .ml file with ocamlfind ocamlc except that we apply cduce --mlstub as a preprocessor A"-pp"; Quote(S[cduce; ..; A"--mlstub"]).

The last rule that I will comment is the one that transform a .intf into a .ml. This rule is particular because it is totally different than the pieces of code of the original Makefile.

Here are the couple of rules that was needed to translate .intf:

examples/adwords/%Service.cmx: examples/adwords/%Service.intf                                                       
   ocamlfind ocamlopt $(OCAMLOPTFLAGS) -c \                                                                          
     -pp "camlp5o ./pa_ocsoapclientstubs.cmo -impl" -c -impl $<

.depend: $(wildcard *.mli) $(wildcard *.ml) \
  $(wildcard examples/adwords/*.mli) $(wildcard examples/adwords/*.ml)
  $(OCAMLDEP) $^ > .depend
  for f in examples/adwords/*.intf; do \
    $(OCAMLDEP) \
    -pp "camlp5o ./pa_ocsoapclientstubs.cmo pr_o.cmo -impl" $$f \
    >> .depend; \
  done

Here is the ocamlbuild rule:

rule "ocsoap: %.intf -> %.ml"
  ~prod:"%.ml"
  ~deps:(if !ocsoap_dev then
           ["%.intf"; !pa_ocsoapclientstubs]
         else
           ["%.intf"])
  begin
    fun env build ->
      Cmd(S[Px !camlp5o; P !pa_ocsoapclientstubs; P "pr_o.cmo"; A"-impl";
            P(env "%.intf"); A"-o"; P(env "%.ml")])
  end
;;

Here we decided to translate directly the .intf into a .ml file. The good thing about ocamlbuild is that it has a powerfull dynamicy dependencies scheme. So here you will generate the .ml file, which in turns will generate a .ml.depends and then will be compiled the standard way. In the makefile, you need to compute the .depends file using a different process that will do everything before the compilation even starts (in fact, before the inclusion of the .depends). We also use the trick to use an OCaml printer (pr_o.cmo) with camlp5o, that will directly output the standard .ml file.

Don't hesitate to post comment if you have question about OASIS and ocamlbuild

Enjoy.

Thursday, June 28 2012

OASIS 0.3.0 release

Logo OASIS small OASIS is a tool to integrate a configure, build and install system in your OCaml project. It helps to create standard entry points in your build system and allows external tools to analyse your project easily. It is the building brick of OASIS-DB, a CPAN for OCaml.

This release takes almost 18 months to complete. This is too long and I will talk in another blog post on the way I am trying to improve this right now (esp. using continuous integration).

This new release fixes a small bug (1 line) that prevents setup.ml to run with OCaml 4.00. If you have projects that was generated with a former release consider upgrading to OASIS 0.3.0.

There are several big new stuff that comes with this release. It now supports Pack: field for libraries which allows to pack your library using -for-pack and so on.

We are also compiling .cmxs (dynlink object for native) by default and we publish them in the META file. This feature was implemented in order to get more libraries to provide .cmxs and to help project like Ocsigen to take advantage of this. If you want to get rid of this at configure time, you can use ./configure --override native_dynlink false.

We introduce two new default flags: --enable-tests and --disable-docs for configure. These are implicit flags that define if we will run Test sections or compile Document sections. They are especially useful to reduce the number of dependencies, because dependencies of Test will be excluded by default. We recommend to set Build$: flag(tests) to any Library or Executable sections that are only useful for tests. This allows you to really cut down the number of dependencies.

The last change I want to introduce is about the old setup-dev subcommand which is now deprecated. It has been replaced by 2 different update schemes. I am pretty excited by this feature which in fact comes from OASIS user (esp. Lwt project). The former scheme was to have a big 'setup.ml' that always call the command oasis to update. This was complex and not very useful. We now have 2 mode: dynamic and weak.

dynamic allow you to have a small setup.ml and to keep your VCS history clean but you need to install OASIS. weak need a big setup.ml but only need to call the command oasis if someone change something in _oasis. This mode is targeted to project that wants to be able to checkout from VCS an OCaml project without installing OASIS. The difference generated by weak mode doesn't pollute too much the VCS history because most the time they make sense. For example, you upgrade your package version number in _oasis and it produces a change with 6 lines where the version number changes in every META, setup.ml files and so on.

This version has been tested on:

  • Linux (Debian Sid and Debian Squeeze with GODI, OCaml 3.12)
  • Windows Vista with MS Visual Studio 2008 toolchain, OCaml 3.12
  • Mac OS X, OCaml 3.12 from GODI

You can download it here or use your favorite package manager: Debian (UPDATE: pending upload)

$ apt-get install -t experimental oasis

GODI

$ godi_console perform -build apps-oasis

odb.ml

$ ocaml odb.ml oasis

Here is the complete changelog:

EXTREMLY IMPORTANT changes (read this)

  • Fix bug with scanf %S@\n for ocaml 4.00. We were unfortunetaly using an undocumented tolerance of Scanf in the previous version. You should consider making new release using this version that fixed this.

PACKAGES uploaded to oasis-db will be automatically "derived" before OCaml 4.00 release (i.e. oUnit v1.1.1 will be regenerated with this new version as oUnit v1.1.1~oasis1).

PACKAGES not uploaded to oasis-db need to be regenerated. In order not to break 3rd party tools that consider a tarball constant, I recommend to create a new version.

Thanks to INRIA OCaml team for synchronizing with us on this point.

Major changes:

  • Handle the field Pack: true to be able to create packed libraries. It also installs .mli files for documentation into the target directory. The pack option is only supported for OASISFormat: 0.3, you will need to update the version of your _oasis file to match it.
  • Introduce --enable-tests to disable tests and docs at oasis level. It seems a very common pattern to have a Flag tests to turn off by default the tests. This is now define as a standard var and you should remove you previous Flag tests but you can continue to use flag(tests) where needed.

You can now have the following example:

     ...
     Executable test_exec
       Install: false
       Build$: flag(tests)
       MainIs: testExec.ml
       BuildDepends: oUnit
     
     Test main
       Command: $test_exec
       TestTools: test_exec
     ...

The Run$: flag(tests) is implicit for the section Test main. The default value is false for tests. If all the executable for test are flagged correctly (Build$: flag(tests)), you'll get rid of the dependency on oUnit.

It works the same for documentation, however the default is true. (Closes: #866)

  • Allow to define interdependent flags

In order to allow interdependent flags, we transform back lazy values into unit -> string functions. This allows to change a flag value on the command line and to update all the dependent values. (Closes: #827, #938)

  • Deprecate the subcommand oasis setup-dev in favor of oasis setup -setup-update {none|weak|dynamic}

It defines different ways to manage the auto-update of setup.ml:

  • none: this is the default mode, and the one you should use when distributing tarballs. No update are performed at al.
  • weak: the update is only triggered when something change in `_oasis`, we keep all files generated
  • dynamic: the content of 'setup.ml' is ultra small (<2kB) and we only keep a small setup.ml, Makefile and configure.

The choice between weak and dynamic depends on your need with regard to VCS and to the presence of oasis. The weak allow to checkout the project from VCS and be able to work on it, without the need of installing 'oasis' as long as you don't change the file _oasis. But it clutter your VCS history with changes to the build system each time you change something in _oasis. The 'dynamic' mode gives you no VCS history pollution but makes mandatory to have installed oasis libraries.

  • Don't copy executable in ocamlbuild

Avoid copying executable to their real name. It helps to call ocamlbuild a single time for the whole build rather than calling it n time (n = number of executable sections) and copying resulting exec.

This speeds up the build process because ocamlbuild doesn't have to compute/scan dependencies each time.

The drawback is that you have to use $foo when you want to call Executable foo, because $foo will be '_build/.../main.byte'.

  • Change the way we parse command line like option (CCOpt, CCLib and the like). We have implemented a real POSIX command line parser, except that variables are processed by Buffer.add_substitute (except if correctly escaped, using Buffer.add_substitute escaping).

For example:

CCOpt: -DEXTERNAL_EXP10 -L/sw/lib "-framework vecLib"

Will be parsed correctly and outputed according to target OS.

  • Minimize the dependencies of the project.

In order to ease building oasis, we have minimize the number of dependencies. You only need to install ocamlmod, ocamlify and ocaml-data-notation for a standard build without tests. Dependencies on pcre, extlib and ocamlgraph has been dropped. The remaining dependencies are hidden behind a flag tests.

  • Implement proposal for handling plugin in META, using "plugin" extra directives

OASIS now produces .cmxs file by default and add them to META.

Now a META looks like:

     ...
     archive(byte) = "oasis.cma"
     archive(byte, plugin) = "oasis.cma"
     archive(native) = "oasis.cmxa"
     archive(native, plugin) = "oasis.cmxs"
     ...

This will ultimately help to generate automatically .cmxs for all oasis enabled projects. We hope that this new feature will improve dynamic linking use in OCaml (esp. for project like Ocsigen).

Other changes

  • Produce only subcommands specific helps when asked (Closes: #783)
  • Use pager for long textual output (Closes: #782)
  • Give the help summary directly when running just "oasis" (Closes: #781)
  • Remove `pwd` filename prefix to avoid too long command line on Windows (Closes: #1007)
  • Install required .cmx when installing libraries (Closes: #686, #885)
  • Retain blanks at the beginning of line in freeform (Closes: #811)
  • Allow PreConfCommand to create setup.data with any variables inside (Closes: #823)
  • Allow to use setup.data in shell script and Makefile (Closes: #982)
  • Fix execute permissions (Closes: #980)
  • Drop extra version in ocaml version (Closes: #964, #888)
  • Always output info messages on stderr (Closes: #785)
  • Fix interversion in version comparator lexer (Closes: #808)
  • Document the fact that setup.data is mandatory (Closes: #946)
  • configure translates getopt command line to Arg (Closes: #778, #804)
  • Generate valid META when there are line breaks in synopsis (Closes: #933)
  • Keep owner of generated files (Closes: #793)
  • Set -I +threads for ocamlfind ocamldoc in ocamlbuild (Closes: #949)
  • Exclude VCS directories by default (Closes: #847)
  • Print the configure summary in the "normal" order.
  • Merge findlib virtual containers with the same name
  • Synchronize plugin data for section
  • Improve documentation
  • Verify validity of input when parsing in quickstart (Closes: #797)
  • Fix stack overflow when there are twice the same library (Closes: #1130)
  • Parse license examples for DEP5 specification (Closes: #1102)
  • Be ultra-liberal regarding version syntax (allow '_' and ' ' in the version).
  • Allow to add extra content to META files (XMETAExtraLines)
  • Allow to use executable name that contains "-" as a variable (replacing "-" by "_").

Thanks to Anil Madhavapeddy, Pierre Chambart, Christophe Troestler, Jeremie Dimino, Ronan Le Hy, Yaron Minsky and Till Varoquaux for their help with this release.

Also thanks to all the testers of the numerous release candidates. This was a long work and each time a tester has downloaded oasis has helped me to know that I was working for someone.

Wednesday, May 2 2012

Installing a new SSD on my laptop

3 years ago I bought a OCZ Core v2 SSD... I was very disappointed by write freeze, price and capacity. Recently a friend of mine, told me that I should try the new ones that has much improved in the meantime. So I decided to give another try to SSD on my laptop. I bought a Crucial M4 128GB SSD, because they seems to be of better quality than OCZ (and probably because of my first failed attempt).

The price is correct (~130€) and after some tests it seems quite good at write and read speed. So from this point of view, I am more convinced than my first attempt.

Now, the real problem of the SSD is how to migrate your old data to your new drive. I have always upgraded my HD to a larger one and so a simple dd if=/dev/sda of=/dev/sdb bs=32M was enough... In the case of SSD you have to migrate to a smaller capacity. This includes to juggle a little bit with your partition to get something that fits.

In order to improve a little bit the lifetime of my SSD, I have tried to follow various advice you can found on the Internet. One of this advice that seems to make sense, is to align your partition to a number modulo 2048 (rather than the default which is to start at 63). I followed this forum to do that.

I end up with all partitions aligned to 2048.

The GOOD news: doing data migration for Linux is a breeze. As a matter of fact, Linux support pretty well being move around, ending up on a different partition and so on... It just took me the time to transfer data, update grub and disk UUID and reboot.

The BAD news: doing data migration for Windows is a hell. I always keep a Windows partition to test various OCaml software. But Windows doesn't like you like to be moved around.

It especially doesn't like that the first sector of its partition get moved (i.e. from sector 63 to sector 2048). I spend a week trying to 'repair windows' but it was a pure waste of time. Thinkpad comes with a special ways to store passwords which is not compatible with the rescue mode of Windows, so you cannot access the repair mode -- because you need the Administrator password -- which is not readable...

I decided to get back to the standard first sector on 63... Now Windows complained about 'hal.dll' missing. You know why ! Because the partition number has changed (from /dev/sda1 to /dev/sda3). Boot in Linux, run fdisk, enter expert mode, fix order -- after having remounted read-only the root partition and unmount everything else. Grub complains a little bit, fall into rescue mode. I fix that following these instructions and running 'grub-install --recheck /dev/sda'.

So after 2 weeks of fights, I end up with a working windows (honestly, this is a pain, don't try to play with Windows partition, this OS is a lot more sensible sensitive than Linux).

Friday, April 27 2012

On the importance of memory buffer...

I have been working since a long time on trying to solve random crashes on the OCaml Forge. I have first suspected failing hardware to be the cause of it, but found no evidence about that.

In despair of a solution, I have installed cacti to investigate the issue. The good news is that it has allowed me to catch a pretty nice graph of a crash.

Load average - Crash

Just before the crash the average load was 200...

My conclusion was that too many processes were running at the same time. So I start to hunt processes that load the server for nothing. One of them was darcsweb. This process doesn't really create a load, but it calls darcs for various operations and most of them are quite expensive. The most expensive one is darcs diff. I first turn on the caching of darcsweb, which already reduce the number of invocation of darcs. But it doesn't make a real difference (except that the website was faster to load).

I continued to investigate. In order to reduce globally the load of the server, I installed various robots.txt to prevent crawler to call diff for all VCS on the forge. Crawler traffic is 10 times the normal traffic and indexing diff in Google is not very interesting. It doesn't make a real difference.

Load average - Peaks

I installed a script to analyze all the peak load above 2. The peaks happen every hour and are more or less important (from 4 to 10). I discovered that this peak were related to the hourly cronjob of FusionForge that fixed repository permissions. I was pretty surprise because updating permissions should no generate this load.

This morning, I had a simple idea. There are a lot of process on the server and one of them (the bzr daemon, loggerhead) eats a big chunk of memory (at least never frees it). I just had a quick at the "buffers" memory... Only 4MB !

I just restarted loggerhead. The improvement made on robots.txt makes it a lot more stable with regard to memory consumption, since we don't run bzr diff anymore. The "buffers" memory is now at 100MB and guess what ! No more peaks...

Here is the result:

Load average - Restart

Conclusion: sometimes the cause of a high load lies in a single number...

Saturday, February 26 2011

OCaml Debian News

... or don't shoot yourself in the foot.

This is not a big secret, Debian Squeeze has been released. Right after this event, the OCaml Debian Task force was back in action -- with Stephane in the leading role. He has planned the transition to OCaml 3.12.0. We will proceed in two steps: a small transition of a reduced set of packages that can be transitioned before 3.12 and then the big transition.

The reason for the small transition, is to avoid having to dep-wait (waiting for dependencies) of package upload by human. In -- a not so far -- past, the OCaml Debian Task force members were uploading packages by hand and waited for a full rebuild to go to the next step. This was long and cumbersome. We use now binNMU: it is binary only uploads -- with no source changes -- processed automatically by the release team and its infrastructure. This is far more effective and helps us to reduce the delay of the transition...

The small transition is happening now!!! Don't update/upgrade your critical Debian installations with OCaml packages, you'll get a lot of removal if you do so. N.B. these removal are part of the famous {{Enforcing type-safe linking using package dependencies}} paper.

As a side note, I am happy to announce that a full round of new OCaml packages has landed in Debian unstable:

People aware of my current work, should notice that all the dependencies of OASIS are now in Debian unstable: ocaml-data-notation, ocamlify, ocaml-expect. This is a hint about the next OCaml Debian package, I will upload. You can also have a look at OASIS enabled packages (all the OASIS dependencies, ocaml-sqlexpr and ocaml-extunix). These packages have been generated using oasis2debian a tool to convert _oasis into debian/ packaging files.

After these transition we will continue proceeding with standard upgrade work (e.g. camomile to 0.8.1).

Sylvain Le Gall is an OCaml consultant working for OCamlCore SARL

Friday, October 22 2010

Compiling pcre-ocaml with Visual Studio 2008

One big change in the recent OASIS v0.2 release is the replacement of Str by Pcre. The big advantage of Pcre is that it can be used in multi-threaded environment, whereas it is not recommended to do so with Str. Since OASIS is used by the OCsigen part of OASIS-DB, we need to make it work safely with Lwt with multiple users at the same time.

Note that, we can probably use Str directly with Lwt, because it is not really multi-threaded. But we want to be safe on this point and Pcre is a very powerful library.

But the OCaml Pcre library depends on an external C library (pcre). This is not a problem on Linux et al, where it is shipped with the OS by default. But on Windows, you need to build it yourself. We want to do it using Microsoft Visual Studio 2008, mainly because OCaml was compiled with it -- and it seems the most natural way to build C library on Windows. As usual, building Open Source C libraries using MS Visual Studio, is not the most common way to process. However, the use of CMake enables a simple way to do it. Here is how we can compile pcre and pcre-ocaml for Windows:

  • Download pcre and unzip it. We use 7.7, which is not the latest version, but the one matching GODI configuration
  • Create c:\pcre\build and c:\pcre\pcre-7.7, beware that you should avoid spaces in the name, there is a small bug in OCaml with spaces for C options
  • Download and install cmake
  • Run cmake-gui and point the sources to the location of your unzipped pcre directory
  • Run Configure a first time
  • Choose Visual Studio 9 2008 generator
  • You should get the configuration display as following:

CMake-GUI with pcre

  • Change the CMAKE_INSTALL_PREFIX to c;\pcre\pcre-7.7, don't select BUILD_SHARED_LIBS
  • Run Configure again
  • Run Generate
  • Start MS Visual Studio 2008
  • Open the PCRE solution located in c;\pcre\build

Visual Studio 2008 with pcre

  • Choose Solution: Release
  • Generate the target ALL_BUILD
  • If you don't have already administrator rights, restart Visual Studio with administrator rights
  • Generate the target INSTALL
  • Go to c:\pcre\pcre-7.7\lib and copy pcre.lib to libpcre.lib, to avoid name clashes with the future OCaml library
  • Download and unzip pcre-ocaml
  • Start a Visual Studio command line and run a cygwin shell inside it (e.g. my script to run cygwin shell with Visual Studio 2008)
  • Change directory to lib directory of the unzipped source of pcre-ocaml
  • Apply this patch, if it is not already applied
  • Edit make_msvc.bat and set the variables PCRE_H and PCRE_LIB to good values set PCRE_H=C:\pcre\pcre-7.7\include and set PCRE_LIB=C:\pcre\pcre-7.7\lib\libpcre.lib
  • Run make_msvc.bat

The best way to test your newly created OCaml pcre library is to try to compile an example from the examples directory of pcre-ocaml itself.

Sylvain Le Gall is an OCaml consultant at OCamlCore SARL

Wednesday, October 20 2010

Unison on windows tips

The big advantage of Unison on Windows is that it allows quite easily to synchronize between Windows and Linux. For those who need to work on Windows with the same set of files as on Linux, this is a big plus. Other tools do it as well, but the 2-way sync of Unison is quite nice. When you need to compile a software on Linux and Windows, you can modify both side at the same time and (almost) don't have problems.

On Windows, the .unison and unison.log are located into your %HOMEPATH%, which the upper directory of the classic Documents folder. In the directory .unison, you will find the .prf files that describe your unison profiles. As usual, default.prf in this directory is the default profile.

Basic

The basic tips are:

  • use fastcheck = true in your default.prf
  • disable directory indexing

Disable directory indexing

You can also disable virus live scan -- if you think it is safe!!!!

SSH

Using ssh under Windows is always a challenge. As a matter of fact, this tool doesn't match Windows context and it is not as integrated as in Linux/BSD.

There is Putty which can help you. It has a good support for remote shell but it is not very easy to setup with Unison. Putty and OpenSSH doesn't have precisely the same set of options and Unison relies on some not available in Putty. There is a script called ssh2plink.bat that can help you using Putty's plink as with Unison. I used it for a while, but this didn't give the expected throughput.

The best option is to use the ssh command provided by Cygwin. In this case you have at the same time good throughput and unison integration. I explain here how to configure you Cygwin's ssh to use a SSH key.

You can bypass the following steps, if you wish to use a password or if you have already setup your ssh to connect to the target computer.

Launch Cygwin's setup.exe and select openssh for installation.

To add a SSH key, launch the cygwin shell:

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
[...]

Copy the file .ssh/id_rsa.pub to your target computer's .ssh/authorized_keys. You should be aware that the file format can be Windows EOL style (in this case use dos2unix to convert the file) and if you copy/paste from a dos box, some end of lines are added and you should remove them from the authorized_keys, to have a single line key.

Once, you have installed your ssh key into target computer, you should try to connect directly from the cygwin shell.

$ ssh XXX

Now, you can configure sshcmd = c:\cygwin\bin\ssh.exe to your default.prf.

Using Cygwin's ssh allow you to get ~2MB/s (or more) when you only get ~100KB/s using ssh2plink.bat.

If you have any other tips to improve Unison on Windows, I will be happy to test them and post it here.

Friday, September 10 2010

Dirty fix for omlet vim extension

omlet (or here) is a vim extension for writing OCaml code.

In my opinion, it has a better indentation than the standard OCaml vim support. Unfortunately, it has a cost: the indentation vim code is more complex. And it has a few bugs :-(

The main bug is that it doesn't like unbalanced comment opening "(*" and closing "*)" tags. From time to time, it enters an infinite (or very long) loop when there such a tag left in your file. It can be very far away from the point you are editing.

It isn't too problematic, because unbalanced tags are a syntax error. But the problem is that it matches these tags inside strings also. So whenever you start using regular expression like "(.*)" the whole indentation fails.

But there is a very ugly solution to this problem!

Problematic code:

 let parse_rgxp =
   Pcre.regexp ~flags:[`CASELESS] 
     "^(?<license>[A-Z0-9\\-]*[A-Z0-9]+)\
      (?<version>-[0-9\\.]+)?(?<later>\\+)?\
      ( *with *(?<exception>.*) *exception)?$"

Solution, add ignore"(*":

 let parse_rgxp =
   ignore "(*";
   Pcre.regexp ~flags:[`CASELESS] 
     "^(?<license>[A-Z0-9\\-]*[A-Z0-9]+)\
      (?<version>-[0-9\\.]+)?(?<later>\\+)?\
      ( *with *(?<exception>.*) *exception)?$"

Very very ugly coder: you balance comment tags in dead code -- very very bad ;-)

ps: another solution when the plugin enters the infinite loop, hit Ctrl-C. This will stop it and let you define your own indentation.

Wednesday, September 1 2010

OCaml 3.12 with Debian Sid right now!

Some careful readers of Planet OCamlCore should wonder why the OCaml packages in Debian has not yet been upgraded to 3.12.0. For the Planet Debian readers, this is the latest version of the Objective Caml programming language.

The answer is simple: Debian Squeeze froze on 6th August. This means that Debian folks focus on fixing release critical bugs and avoid doing big transitions in unstable (Sid). In particular, the Debian OCaml maintainers has decided to keep OCaml 3.11.2 for Squeeze, because the delay was really too short: OCaml 3.12 was out on 2nd August.

A great work has already been done by S. Glondu and the rest of the Debian OCaml maintainers to spot possible problems. The result was a series of bugs submitted to the Debian BTS. This effort has started quite early and have been updated with various OCaml release candidates.

S. Glondu has also built an unofficial Debian repository of OCaml 3.12.0 packages here.

Let's use it to experiment with OCaml 3.12.0.

schroot setup

Following my last post about schroot and CentOS, we will use a schroot to isolate our installation of unofficial OCaml 3.12.0 packages.

approx

approx is a debian caching proxy server for Debian archive files. It is very effective and simple to setup. It is already on my server (Debian Lenny, approx v3.3.0). I just have to add a single line to create a proxy for ocaml 3.12 packages:

 $ echo "ocaml-312   http://ocaml.debian.net/debian/ocaml-3.12.0" >> /etc/approx/approx.conf
 $ invoke-rc.d approx restart

approx is written in OCaml, if you want to know how I come to it.

debootstrap and schroot

We create a chroot environment with Debian Sid:

# PROXY = host where approx is installed, debian/ points to official Debian repository of 
# your choice. 
$ debootstrap sid sid-amd64-ocaml312 http://PROXY:9999/debian

We create a section for sid-amd64-ocaml312 in /etc/schroot/schroot.conf (Debian Lenny):

[sid-amd64-ocaml312]
description=Debian sid/amd64 with OCaml 3.12.0
type=directory
location=/srv/chroot/sid-amd64-ocaml312
priority=3
users=XXX
root-groups=root
run-setup-scripts=true
run-exec-scripts=true

Replace XXX by your login.

And we install additional softwares:

 $ schroot -c sid-amd64-ocaml312 apt-get update
 $ schroot -c sid-amd64-ocaml312 apt-get install vim-nox sudo

OCaml 3.12 packages

Now we can start the setup to access OCaml 3.12.0 packages.

The repository is signed by S. Glondu GPG key (see here). We need to get it and inject it into apt:

$ gpg --recv-key 49881AD3 
gpg: requête de la clé 49881AD3 du serveur hkp keys.gnupg.net
gpg: clé 49881AD3: « Stéphane Glondu <steph@glondu.net> » n'a pas changé
gpg:        Quantité totale traitée: 1
gpg:                      inchangée: 1
$ gpg -a --export 49881AD3 > glondu.gpg
$ schroot -c sid-amd64-ocaml312 apt-key add glondu.gpg

The following part is done in the schroot:

$ schroot -c sid-amd64-ocaml312
# PROXY = host where approx is installed
(sid-amd64-ocaml312)$ echo "deb http://PROXY:9999/ocaml-312 sid main" >> /etc/apt/sources.list
(sid-amd64-ocaml312)$ cat <<EOF >> /etc/apt/preferences
Package: *
Pin: release l=ocaml
Pin-Priority: 1001
EOF
(sid-amd64-ocaml312)$ apt-get update 
...
(sid-amd64-ocaml312)$ apt-cache policy ocaml
  Installé : (aucun)
  Candidat : 3.12.0-1~38
 Table de version :
     3.12.0-1~38 0
       1001 http://atto/ocaml-312/ sid/main amd64 Packages
     3.11.2-1 0
        500 http://atto/debian/ sid/main amd64 Packages
(sid-amd64-ocaml312)$ apt-get install ocaml-nox libtype-conv-camlp4-dev libounit-ocaml-dev...

That's it. The apt-policy command shows that OCaml 3.12 for the ocaml-312 repository has an higher priority for installation.

Good luck playing with OCaml 3.12.0.

Thursday, August 26 2010

CentOS 5 chroot with schroot

OCaml compiles native executables in static mode. It allows to have a minimal set of dependencies when delivering an executable. It has also disadvantages like the size of the executable and problems arising when considering libraries update -- but this is another topic. There is still one strong dependency that you should not forget when you want to deliver a product for most of the Linux distributions: dependency on the glibc version.

Trying to run OASIS compiled with Debian Lenny, on CentOS 5.5:

$ OASIS
.../OASIS: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by .../OASIS)

So when compiling for delivery, one should choose the oldest distribution he targets. In my case, I choose CentOS 5 which comes with glibc v2.5. I usually choose Debian stable at the moment of writing Debian Lenny. But for now, the Debian Lenny's glibc is newer (v2.7) than the one coming from the CentOS 5.5 stable release. CentOS is a Red Hat like Linux distribution.

I use a Debian Lenny amd64 host system and I decided to setup a chroot of CentOS 5 i386 and amd64. I also setup schroot to use my CentOS chroot.

CentOS 5 amd64 setup

First of all we use rinse, which can setup a RPM based distribution in a chroot. The version v1.3 shipped with Debian Lenny has some bugs: it doesn't install nss and other mandatory packages. So I downloaded v1.7 directly from Debian Sid. There is no dependencies problems and the package is arch:all, so it is straightforward to install:

$ wget http://ftp.de.debian.org/debian/pool/main/r/rinse/rinse_1.7-1_all.deb # Replace ftp.de.debian.org by your preferred Debian mirror
$ dpkg -i rinse_1.7-1_all.deb

Then I create the chroot directory and launch rinse:

$ mkdir /srv/chroot/centos5-amd64
$ rinse --arch amd64 --distribution centos-5 --directory /srv/chroot/centos5-amd64 # N.B. you must use --arch, the default is i386

Once installation is complete, you can add an entry for this distribution in /etc/schroot/schroot.conf:

[centos5-amd64]
description=Centos 5 (amd64)
location=/srv/chroot/centos5-amd64
priority=3
users=XXX
groups=
root-groups=root
type=directory
run-setup-scripts=true
run-exec-scripts=true

Replace XXX by your login.

If you try to login directly, you will get warnings:

$ schroot -c centos5-i386
I : [chroot centos5-i386-a952de23-7f4b-4bae-a9b9-752ecee4a185] Exécution de l'interpréteur de commandes initial : « /bin/bash »
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied

This is a bit misleading because the real problem is that nothing is created in /dev/. CentOS delegates creating char/block devices to udev. You have two solutions to solve this issue:

  • login and call MAKEDEV to create missing devices:
$ MAKEDEV random
$ MAKEDEV console
$ MAKEDEV zero
$ MAKEDEV null
$ MAKEDEV stdout
$ MAKEDEV stdin
$ MAKEDEV stderr
  • use an already setup Debian chroot to copy the missing devices:
$ rsync -av /srv/chroot/lenny-amd64/dev/* /srv/chroot/centos5-amd64/dev/

That's it, you now have a functional chrooted CentOS 5 environment:

$ schroot -c centos5-amd64 cat /etc/redhat-release
I : [chroot centos5-amd64-b9bae264-285b-4d17-a046-13386736cecd] Exécution de la commande : « cat /etc/redhat-release »
CentOS release 5.5 (Final)

CentOS 5 i386 setup

To setup an i386 environment, we follow almost the same scheme, except we need to fix a bug in rinse v1.7: we need to call linux32 before executing chroot. The problem is that the first stage installation of rinse install an i386/686 environment but as soon as you call chroot yum install ..., it will guess that the system is amd64 and will install missing packages. See the Debian bug report and the example patch attached to correct this behavior.

WARNING: this patch is just an example, you can apply it for creating CentOS i386 chroot on Lenny amd64 host but you should remove the patch as soon as the installation is complete.

$ mkdir /srv/chroot/centos5-i386/
$ rinse --arch i386 --distribution centos-5 --directory /srv/chroot/centos5-i386 # With /usr/lib/rinse/centos-5/post-install.sh patched 
$ rsync -av /srv/chroot/lenny-i386/dev/* /srv/chroot/centos5-i386/dev/

Add this distribution to /etc/schroot/schroot.conf:

[centos5-i386]
description=Centos 5 (i386)
location=/srv/chroot/centos5-i386
priority=3
users=XXX
groups=
root-groups=root
type=directory
run-setup-scripts=true
run-exec-scripts=true
personality=linux32

You now have a schroot of CentOS 5 i386:

$ schroot -c centos5-i386 cat /etc/redhat-release
I : [chroot centos5-i386-9acafa91-9862-4488-aaef-4ab2a482771e] Exécution de la commande : « cat /etc/redhat-release »
CentOS release 5.5 (Final)

Happy schroot hacking!

Monday, August 2 2010

OCaml 3.12.0 is out: watch the movie

I have been quite busy the few last months. But anyway, I found time and solved various technical pitfalls to be able to bring you the first movie of the OCaml Meeting:

Foreword by X. Leroy at OCaml Meeting 2010 (subtitle: OCaml 3.12.0 features presentation)

In this video, Xavier Leroy told us about the features in OCaml 3.12.0. This version is now released so it is high time to release the matching movie.

I will release other movies of the OCaml Meeting during August and will try to explain the various pitfalls I encounter -- and the OCaml solutions I use to solve them.

Wednesday, July 14 2010

Project children has forked

The new project shares the same code base, but this branch is a total rewrite of the project children.

Clementine is born on 8th July 2010, at 12:01 Paris time. She is now at home with her mother, father and brother. She is very peaceful and hardly cry once or twice in a day: at bath time and around 5am for her night lunch.

Clementine by Bernard Le Gall

I will probably lack of time in the coming month. Expect some delays with the OCaml forge, other OCaml projects I maintain and my OCaml Debian packages.

Thursday, July 1 2010

5 years old CD-RW...

This blog post is a kind of followup of a previous blog post about reading 10 years old CD-R.

Today, I get a new shiny ASRock ION 330 to play with, in itself this PC has nothing particular. As usual, I begin by using System Rescue CD to check bad blocks on the hard drive, before working with it (badblocks -v -s -w -c 4096 /dev/sda). Once done, I will check the memory (memtest). This is my standard procedure when receiving a new PC.

But I don't have my System Rescue CD at hand. No problem, I download it and and write it on an old CD-RW, bought 5 years ago but still sealed into his blister. For your information, this is a Memorex CD-RW 700MB 16-24x. I blank it (wodim blank=all). I write it at speed 16x, because I don't know how to lower the speed. After writing the CD, I re-read it to check everything is fine (readom dev=/dev/cdrw1 f=test.iso).

Ooops:

$ readom dev=/dev/cdrw1 f=test.iso
[...] 
Errno: 5 (Input/output error), read_g1 scsi sendcmd: no error
CDB:  28 00 00 01 3C 00 00 00 40 00
status: 0x2 (CHECK CONDITION)
Sense Bytes: 70 00 03 00 00 00 00 0A 00 00 00 00 11 05 00 00
Sense Key: 0x3 Medium Error, Segment 0
Sense Code: 0x11 Qual 0x05 (l-ec uncorrectable error) Fru 0x0
Sense flags: Blk 0 (not valid) 
cmd finished after 6.614s timeout 40s
readom: Input/output error. Cannot read source disk
readom: Retrying from sector 80896.
.............
readom: Error on sector 80908 corrected after 9 tries. Total of 0 errors.
..............---

OK, I re-blank it (wodim blank=fast) and rewrite it. The CD seems to be readable this time. But when I try to boot with it my new ASRock, it fails.

Enough is enough, I get an heavy used Verbatim 8cm CD-RW. Just to show you how old he is: the first ISO written on it was "Debian Sarge". And the magic happens: it works on the first try!!!!

You will probably think that I work for Verbatim. This is not true but I must admit that first-price CD-RW/CD-R seems to be very low quality and don't hold data in time. This is probably related to the chemical compounds of the CD. The failing Memorex CD-RW will end up in a trash and I will continue to use my 5 years old Verbatim CD-RW.

Dear reader, in order to find a way not to dump this failing CD-RW, do you know a way to make it to work?

What you should now about the failing CD-RW:

  • there are no scratches on the surface
  • the CD-RW has never been used before this morning
  • I keep it out of the direct sunlight
  • it was stored next to the Verbatim CD-RW

Thursday, June 10 2010

Waiting for her in ~1month

My wife is pregnant and we are expecting our second baby's arrival in about a month. Last time, she came back from her preparation lessons with a adverstisement "baby shower" gift pack.

One of them catch my attention:

English translation: be prepared to offer him the best...

English translation: because Debian guarantee the quality and offering the quality is a proof of love

OK, the swirl is the other way and I made a 5 minutes GIMP modification to cut-and-paste the Debian logo. But the message is here!

Remember me:

For those interested, the real thing is a soap called NUK(R).

- page 1 of 4