Hello,
I've ported Babel's small test suite from RT to Stefil and consequently I have some fixes to contribute, some questions and some suggestions.
I've pushed the small, trivial fixes to http://common-lisp.net/~loliveira/darcs/stefil/. I suppose having *test-progress-print-right-margin* default to 80 might be controversial with you guys. I'd argue that that's the default width for a lot of applications (xterm, emacs, etc) but the truth is that I'm an 80-column freak, really. Feel free to ignore the patches you disagree with.
Here are a few suggestions for Stefil. If you think they're worthwhile, I'll prepare proper patches. However, I suspect there might be better, more idiomatic solutions for some of these problems within the existing framework that I'm unaware of.
;; example: (returns (string-size-in-octets "abc" :encoding :ascii) 3 3) (defmacro returns (form &rest values) "Asserts, through EQUALP, that FORM returns VALUES." `(is (equalp (multiple-value-list ,form) (list ,@values))))
(defun fail (control-string &rest arguments) (stefil::record-failure 'stefil::failed-assertion :format-control control-string :format-arguments arguments))
(defun expected (expected &key got) (fail "expected ~A, got ~A instead" expected got))
Here's an example of how I use EXPECTED/FAIL:
(deftest dec.ascii.2 () (handler-case (octets-to-string (ub8v 97 128 99) :encoding :ascii :errorp t) (character-decoding-error (c) (is (equalp #(128) (character-decoding-error-octets c))) (is (eql 1 (character-coding-error-position c))) (is (eq :ascii (character-coding-error-encoding c)))) (:no-error (result) (expected 'character-decoding-error :got result))))
Another idea would be to use SIGNALS:
(let ((c (signals (octets-to-string ...)))) ;; test C )
However, it seems to me that this would introduce bogus failures and/or errors when SIGNALS returns NIL that would make debugging more difficult. Any suggestions?
Since babel-tests was using RT, it had lots of small tests like this: (deftest foo form retval1 retval2 ...) so I found myself doing a lot of (deftest foo () (is (equalp form retval1))) and worse for multiple return values. Before writing the RETURNS macro I wrote a DEFSTEST macro that works like RT's:
(defmacro defstest (name form &body return-values) "Similar to RT's DEFTEST." `(deftest ,name () (returns ,form ,@(mapcar (lambda (x) `',x) return-values))))
I like it because I'm used to RT and it made porting most of these small tests a simple matter of replacing DEFTEST with DEFSTEST but I'm not sure how useful it'd be in general.
There a few itches I'd like to scratch before porting other test suites I maintain. Here's my TODO list, let me know if you have any objections or better alternatives to these features:
* make FUNCALL-TEST-WITH-FEEDBACK-MESSAGE print a list of failed tests (perhaps along with descriptions of the failed assertions). This could be useful for bug reports, buildbots, and other non-interactive use cases.
* implement a functional interface for marking expected failures. E.g.: #+lispworks (expected-failure 'some-test-lispworks-does-not-like).
* implement a way of running tests through EVAL, instead of compiling. We use this in cffi-tests to test whether removing compiler-macros introduces bugs and stuff like that.
* minor thing: export STEFIL::ROOT-SUITE so that one can write (deftest (foo :in root-suite)).
Finally, reducing the dependencies to external libraries would help with lobbying for porting cffi-tests (and others) to Stefil. Would that be in any way acceptable for you guys?
Thanks for Stefil and sorry for the long email.