Hi,
Some people may know that Juho Snellman committed a nice new tool to SBCL recently: a tool to measure code coverage[1]. For laughs, I thought I'd try it out on McCLIM and its test suite...
To do so, I first loaded clx, spatial-trees and flexichain. Then (require :sb-cover) (declaim (optimize sb-cover:store-coverage-data)) (asdf:oos 'asdf:load-op :mcclim) (sb-cover:reset-coverage) ; debatable. See below. (defpackage :clim-tests (:use :clim :clim-lisp)) (mapcar #'load (directory "/path/to/mcclim/Tests/*.lisp"))
The results I got are available at http://doc.gold.ac.uk/~mas01cr/tmp/mcclim-tests/.
So, firstly, this is clearly not automated enough; I plan to spend a little time making running tests easier. The point about clearing coverage before running the tests is to remove all the uses of internal code (such as macros) in the building of mcclim itself. I don't know whether that's fair or not, but in doing so I noticed that there are some load-time tests (in e.g. bezier.lisp) which might be better placed in a test suite.
I also am aware that I didn't run Drei or automaton test suites in the process of doing this; running those should be a simple matter.
What does this show? Well, even though it's difficult to test a UI library in an automated fashion, I'm pretty sure that there are large swathes of things that /are/ testable in an automated fashion that aren't being so tested -- and that maybe by making these statistics available people could find bits that interest them and write tests to exercise those bits.
There's also the obvious possibility of collecting the coverage statistics while running demodemo -- which will exercise backend functions and some of the UI code, and show us where some gaps are. (There's also my long-standing idea of using the Null backend to test UI in an automated fashion, but I'm sure more work needs to be done to make that possible.)
Anyway. Comments, ideas?
Cheers,
Christophe
[1] Weblog entry at http://jsnell.iki.fi/blog/archive/2007-05-03-code-coverage-tool-for-sbcl.html; some details at http://www.sbcl.org/manual/sb_002dcover.html.