Nikodemus Siivola wrote:
2009/8/5 Robert Goldman rpgoldman@sift.info:
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
I suppose, but typically a regression test suite should attempt to run ALL of its tests, collecting a set of failing tests, and then returning that information. And demanding interactive attention ("I'm going to throw you into the debugger if I fail") may also be undesirable in practice. I suppose handler-case would be enough for that, but I'm still uncomfortable with the idea.
My rationale for suggesting this was that there are two mental modes for running tests: as a user ("Do I have version that seems to be working?") and as a developer ("What did I break now?").
I would expect TEST-SYSTEM to be used mostly by the end users of ASDF systems, whereas presumably developers run their own test-suites more directly -- I'm assuming that the typical TEST-OP definition would look like this:
(defmethod perform ((op test-op) (sys (eql (find-system :my-system)))) (foo-unit:run-tests :my-system-tests :on-failure :error))
and developers of MY-SYSTEM would normally use RUN-TESTS directly, taking advantage of whatever reporting and debugging features the test-suite provides.
I see. However, I think it would be a mistake to force /anyone/ to encounter an immediate error on running the test-op as in the snippet above.
I work on a couple of large systems where, in fact, the developers run their own test suites through ASDF, because many of them are not familiar with the test tool (maybe they are working on the java side). They don't want to encounter the lisp debugger. For that matter, we actually package up the results of the tests into an XML file that can be slurped by junit, and assembled into an overall test result. In fact, in some cases there isn't even someone available to interact with the build process --- it's run at a high level by scripts.
For relatively small libraries, doing one thing, the "error immediately" behavior might be appropriate. But for large, complex artifacts (e.g., a web site), this behavior is not appropriate. It would be appropriate to /offer/ error-immediately behavior, but not to offer /only/ that behavior.
Best, Robert
Best, R