2009/8/5 Robert Goldman rpgoldman@sift.info:
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
I suppose, but typically a regression test suite should attempt to run ALL of its tests, collecting a set of failing tests, and then returning that information. And demanding interactive attention ("I'm going to throw you into the debugger if I fail") may also be undesirable in practice. I suppose handler-case would be enough for that, but I'm still uncomfortable with the idea.
My rationale for suggesting this was that there are two mental modes for running tests: as a user ("Do I have version that seems to be working?") and as a developer ("What did I break now?").
I would expect TEST-SYSTEM to be used mostly by the end users of ASDF systems, whereas presumably developers run their own test-suites more directly -- I'm assuming that the typical TEST-OP definition would look like this:
(defmethod perform ((op test-op) (sys (eql (find-system :my-system)))) (foo-unit:run-tests :my-system-tests :on-failure :error))
and developers of MY-SYSTEM would normally use RUN-TESTS directly, taking advantage of whatever reporting and debugging features the test-suite provides.
Cheers,
-- Nikodemus