Blog of Sylvain Le Gall

After a long pause, I have resumed my work on OUnit2. It is going quite well.

The basic features:

• better configuration setup
  • environment variable
  • command line options
  • configuration files
• systematic logging (verbose always on), but output log in a file
• allow vim quickfix to jump in the log file where the error has happened
• output HTML report
• output JUnit report
• choose how to run your test:
  • run tests in parallel using processes (auto-detect number of CPU and run as many worker processes)
  • run tests concurrently using threads
  • use the old sequential runner
• refactoring of the bracket, now easier to use
• refactoring of OUnit2 interface (remove all useless functions)
• non-fatal section: allow to fail inside non-fatal section without quitting immediately the whole test
• allow to use OUnit1 test inside OUnit2 (smooth the transition)

I still need to do the following:

• a test chooser that does smart selection of tests:
  • run the one that failed in the last run and run them first
  • before re-running the one that was ok, check that all failing tests are now passing otherwise skip the already passing tests.
• timer that makes tests fail if they take too long
• allow to parametrize output filename so that you can use OUNIT_OUTPUT_FILE=ounit-$(name),log and have $(name) replace by the test suite name
• create locks to avoid accessing the same resources within a single process or the whole application (typically to avoid doing a chdir while another thread is doing a chdir elsewhere)
• better logging when using multiple workers
• add more tests for the new runners
• introduce a 'cached' state to avoid rerunning a test if you can programmaticaly determine that the result will be the same.
• create a in_testdata_dir function to locate test data, if any
• sys admin (website, release process)
• update the whole documentation

There is still a lot of work, but the current results are already quite good. The speed improvement of the processes runner is a good thing to shorten test timing (HINT: tester needed!).

Focus on: the new bracket.

In OUnit 1, a bracket was very functional:

 bracket 
    (fun () -> "foo")  (* setup *)
    (fun foo -> ())
    (fun foo -> ())  (* tear down *)

So for common bracket, like bracket_tmpfile

   bracket_tmpfile
       (fun (fn, chn) ->
             (* Do something with chn and fn *)

The problem is that if you were using 2 or 3 temporary files, the level of indentation was high. I have decide to switch to a more imperative approach, registering the tear down function inside the test context:

   let (fn1, chn1) = bracket_tmpfile ctxt in
   let (fn2, chn2) = bracket_tmpfile ctxt in
      ....

This is shorter and clearer (albeit less functional).

Focus on: non fatal section

Sometimes, you want to verify a set of properties but to have a clear vision of what is going wrong, you need to do more than one assert.

In OUnit1, you can do:

 assert_equal exp1 v1;
 assert_equal exp2 v2

But if exp1 <> v1, you quit immediately and you'll never know for exp2 and v2.

In OUnit2, you can do:

 non_fatal ctxt (fun ctxt -> assert_equal exp1 v1);
 non_fatal ctxt (fun ctxt -> assert_equal exp2 v2)

In this new version, you will test both equality and the result of the test will be the worst failure you get (or success if both of them succeed).

Help wanted

If you have a long standing issue with OUnit, this is the time to submit a bug. OUnit BTS

If you want to try the dev version of OUnit:

 $> darcs get http://forge.ocamlcore.org/anonscm/darcs/ounit/ounit

Patches always welcome.

Special thanks to Thomas Wickham who has entirely written OUnitRunnerThreads and kickstarted the processes runner.