Sorry to parasite this mailing-list, but I'm trying to design the testing aspect of XCVB right now. The constraint is that we want to be able to use make as a backend, and that test results should thus be reified as files if we want to avoid re-running already-computed tests.
My dim ideas about it include the following: * test reports are targets depending on properly compiled fasls or images, produce test reports. * a test report is a file the first form of which is a simple SEXP to be read, following some standardized structure to specify overall success, status of individual tests (including error message, maybe information as to last success, etc.). * from the test reports, a success witness (empty file) may be created on success. * the success witness fails to be created when the test wasn't a success, triggering an error for make to catch. * preparing the makefile erases the success witnesses.
If ASDF's test-op is to be standardized, may I suggest that we agree on the format for a SEXP to be returned, that will report the test results?
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] A child of five would understand this. Send someone to fetch a child of five. -- Groucho Marx
2009/8/5 Robert Goldman rpgoldman@sift.info:
Nikodemus Siivola wrote:
2009/8/5 Robert Goldman rpgoldman@sift.info:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
Cheers,
-- Nikodemus
A second point is that test libraries are often very chatty, and it's unpleasant to have their output mixed in the same stream with the output of the compiler and whatever all else ASDF might happen to trigger splatting onto the output stream. So I really think that some stream argument is essential, even if we were to accept your proposal.
best, r