Here are a few suggestions for Stefil. If you think they're worthwhile, I'll prepare proper patches. However, I suspect there might be better, more idiomatic solutions for some of these problems within the existing framework that I'm unaware of.

;; example: (returns (string-size-in-octets "abc" :encoding :ascii) 3 3) (defmacro returns (form &rest values) "Asserts, through EQUALP, that FORM returns VALUES." `(is (equalp (multiple-value-list ,form) (list ,@values))))

although we've never needed it, this looks generally useful.

(defun fail (control-string &rest arguments) (stefil::record-failure 'stefil::failed-assertion :format-control control-string :format-arguments arguments))

(defun expected (expected &key got) (fail "expected ~A, got ~A instead" expected got))

i'm not sure about these. they look useful, but i'm a bit reluctant in growing the api too much. but...

Here's an example of how I use EXPECTED/FAIL:

(deftest dec.ascii.2 () (handler-case (octets-to-string (ub8v 97 128 99) :encoding :ascii :errorp t) (character-decoding-error (c) (is (equalp #(128) (character-decoding-error-octets c))) (is (eql 1 (character-coding-error-position c))) (is (eq :ascii (character-coding-error-encoding c)))) (:no-error (result) (expected 'character-decoding-error :got result))))

OTOH, this looks like a good reason for them and it would be hard to do this using IS.

Another idea would be to use SIGNALS:

(let ((c (signals (octets-to-string ...)))) ;; test C )

However, it seems to me that this would introduce bogus failures and/or errors when SIGNALS returns NIL that would make debugging more difficult. Any suggestions?

(let ((c (signals (octets-to-string ...)))) (when c ;; test c )) which doesn't look too good.

Since babel-tests was using RT, it had lots of small tests like this: (deftest foo form retval1 retval2 ...) so I found myself doing a lot of (deftest foo () (is (equalp form retval1))) and worse for multiple return values. Before writing the RETURNS macro I wrote a DEFSTEST macro that works like RT's:

(defmacro defstest (name form &body return-values) "Similar to RT's DEFTEST." `(deftest ,name () (returns ,form ,@(mapcar (lambda (x) `',x) return-values))))

I like it because I'm used to RT and it made porting most of these small tests a simple matter of replacing DEFTEST with DEFSTEST but I'm not sure how useful it'd be in general.

i think it's trivial enough to let the users do it for themselves. we have several different deftest variants in the various libs anyway.

There a few itches I'd like to scratch before porting other test suites I maintain. Here's my TODO list, let me know if you have any objections or better alternatives to these features:

* make FUNCALL-TEST-WITH-FEEDBACK-MESSAGE print a list of failed tests (perhaps along with descriptions of the failed assertions). This could be useful for bug reports, buildbots, and other non-interactive use cases.

sounds good! the original reason for that function was that asdf does not return anything from the operations, so you can't communicate back the results normally when running an 'asdf:test-op.

* implement a functional interface for marking expected failures. E.g.: #+lispworks (expected-failure 'some-test-lispworks-does-not-like).

i think this would be non-trivial (iow, fragile) to implement in the current setup and as speed does not matter here, maybe i'd simply add a (defmacro with-expected-failures* (expression &body body) "Any failure inside the dynamic extent of this block is registered as an expected failure when the expression evaluates to T." `(bind ((*failures-and-errors-are-expected* (if ,expression #t *failures-and-errors-are-expected*))) ,@body)) (with-expected-failures* (featurep :lispworks) ...)

* implement a way of running tests through EVAL, instead of compiling. We use this in cffi-tests to test whether removing compiler-macros introduces bugs and stuff like that.

i'm open for this.

* minor thing: export STEFIL::ROOT-SUITE so that one can write (deftest (foo :in root-suite)).

done.

Finally, reducing the dependencies to external libraries would help with lobbying for porting cffi-tests (and others) to Stefil. Would that be in any way acceptable for you guys?

i got rid of :defclass-star and i'm open for getting rid of :iterate (although i always regret that later... :) but alexandria and bind are pretty much by-default dependencies for us nowadays.

Thanks for Stefil and sorry for the long email.

thanks for being the first serious external user with actual feedback! :) -- attila