Faré wrote:
Sorry to parasite this mailing-list, but I'm trying to design the testing aspect of XCVB right now. The constraint is that we want to be able to use make as a backend, and that test results should thus be reified as files if we want to avoid re-running already-computed tests.
My dim ideas about it include the following:
- test reports are targets depending on properly compiled fasls or images,
produce test reports.
- a test report is a file the first form of which is a simple SEXP to
be read, following some standardized structure to specify overall success, status of individual tests (including error message, maybe information as to last success, etc.).
- from the test reports, a success witness (empty file) may be created
on success.
- the success witness fails to be created when the test wasn't a
success, triggering an error for make to catch.
- preparing the makefile erases the success witnesses.
If ASDF's test-op is to be standardized, may I suggest that we agree on the format for a SEXP to be returned, that will report the test results?
Based on our experience in a large, multi-language framework, I'd suggest a related enhancement:
* specify a translation from the s-expression test result format to XML that complies with what junit expects.
In collaboration with folks from BBN, my colleague John Maraist has built this capability into his NST test framework. This means that a large system with java components can all be tested together.
best, r