Hi,
Some people may know that Juho Snellman committed a nice new tool to SBCL recently: a tool to measure code coverage[1]. For laughs, I thought I'd try it out on McCLIM and its test suite...
To do so, I first loaded clx, spatial-trees and flexichain. Then (require :sb-cover) (declaim (optimize sb-cover:store-coverage-data)) (asdf:oos 'asdf:load-op :mcclim) (sb-cover:reset-coverage) ; debatable. See below. (defpackage :clim-tests (:use :clim :clim-lisp)) (mapcar #'load (directory "/path/to/mcclim/Tests/*.lisp"))
The results I got are available at http://doc.gold.ac.uk/~mas01cr/tmp/mcclim-tests/.
So, firstly, this is clearly not automated enough; I plan to spend a little time making running tests easier. The point about clearing coverage before running the tests is to remove all the uses of internal code (such as macros) in the building of mcclim itself. I don't know whether that's fair or not, but in doing so I noticed that there are some load-time tests (in e.g. bezier.lisp) which might be better placed in a test suite.
I also am aware that I didn't run Drei or automaton test suites in the process of doing this; running those should be a simple matter.
What does this show? Well, even though it's difficult to test a UI library in an automated fashion, I'm pretty sure that there are large swathes of things that /are/ testable in an automated fashion that aren't being so tested -- and that maybe by making these statistics available people could find bits that interest them and write tests to exercise those bits.
There's also the obvious possibility of collecting the coverage statistics while running demodemo -- which will exercise backend functions and some of the UI code, and show us where some gaps are. (There's also my long-standing idea of using the Null backend to test UI in an automated fashion, but I'm sure more work needs to be done to make that possible.)
Anyway. Comments, ideas?
Cheers,
Christophe
[1] Weblog entry at http://jsnell.iki.fi/blog/archive/2007-05-03-code-coverage-tool-for-sbcl.html; some details at http://www.sbcl.org/manual/sb_002dcover.html.
Christophe Rhodes csr21@cantab.net writes:
There's also the obvious possibility of collecting the coverage statistics while running demodemo -- which will exercise backend functions and some of the UI code, and show us where some gaps are.
I've now done this: the results of compiling and loading the files in the mcclim/ and mcclim/Backends/CLX/ directories, and then starting demodemo and interacting a little bit with each of the demos, is up at http://www-jcsu.jesus.cam.ac.uk/~csr21/tmp/mcclim-tests/.
Cheers,
Christophe
Christophe Rhodes wrote:
Hi,
Some people may know that Juho Snellman committed a nice new tool to SBCL recently: a tool to measure code coverage[1]. For laughs, I thought I'd try it out on McCLIM and its test suite...
To do so, I first loaded clx, spatial-trees and flexichain. Then (require :sb-cover) (declaim (optimize sb-cover:store-coverage-data)) (asdf:oos 'asdf:load-op :mcclim) (sb-cover:reset-coverage) ; debatable. See below. (defpackage :clim-tests (:use :clim :clim-lisp)) (mapcar #'load (directory "/path/to/mcclim/Tests/*.lisp"))
The results I got are available at http://doc.gold.ac.uk/~mas01cr/tmp/mcclim-tests/.
So, firstly, this is clearly not automated enough; I plan to spend a little time making running tests easier. The point about clearing coverage before running the tests is to remove all the uses of internal code (such as macros) in the building of mcclim itself. I don't know whether that's fair or not, but in doing so I noticed that there are some load-time tests (in e.g. bezier.lisp) which might be better placed in a test suite.
I also am aware that I didn't run Drei or automaton test suites in the process of doing this; running those should be a simple matter.
What does this show? Well, even though it's difficult to test a UI library in an automated fashion, I'm pretty sure that there are large swathes of things that /are/ testable in an automated fashion that aren't being so tested -- and that maybe by making these statistics available people could find bits that interest them and write tests to exercise those bits.
There's also the obvious possibility of collecting the coverage statistics while running demodemo -- which will exercise backend functions and some of the UI code, and show us where some gaps are. (There's also my long-standing idea of using the Null backend to test UI in an automated fashion, but I'm sure more work needs to be done to make that possible.)
Anyway. Comments, ideas?
These results are very thought provoking, but I was wondering --- is there a source of information about what the counts mean somewhere? I have looked at the sb-cover manual section and Juho's page, and I don't see a definition for these terms. I am asking because I'm trying to grope towards an understanding of how to use them. For example, I see in your statistics that graph-formatting.lisp (one of the few pieces of McCLIM I can claim to somewhat understand), has 54.2% expression coverage and 46.8% branch coverage. I'm not sure how to relate that to what I know, which is that graph-formatting is radically unfinished (it supports only tree formatting, neither DAG nor directed graph formatting options, at least as of about a month ago, which was when I last pulled a CVS update).
Thanks, r
-
Christophe Rhodes -
Robert Goldman