stefil-devel

Download

stefil-devel@common-lisp.net

June 2008

1 participants
1 discussions

[stefil-devel] First impressions after porting a small test suite from RT to Stefil
by Luís Oliveira 22 Jun '08

22 Jun '08

Hello, I've ported Babel's small test suite from RT to Stefil and consequently I have some fixes to contribute, some questions and some suggestions. I've pushed the small, trivial fixes to <http://common-lisp.net/~loliveira/darcs/stefil/>. I suppose having *test-progress-print-right-margin* default to 80 might be controversial with you guys. I'd argue that that's the default width for a lot of applications (xterm, emacs, etc) but the truth is that I'm an 80-column freak, really. Feel free to ignore the patches you disagree with. Here are a few suggestions for Stefil. If you think they're worthwhile, I'll prepare proper patches. However, I suspect there might be better, more idiomatic solutions for some of these problems within the existing framework that I'm unaware of. ;; example: (returns (string-size-in-octets "abc" :encoding :ascii) 3 3) (defmacro returns (form &rest values) "Asserts, through EQUALP, that FORM returns VALUES." `(is (equalp (multiple-value-list ,form) (list ,@values)))) (defun fail (control-string &rest arguments) (stefil::record-failure 'stefil::failed-assertion :format-control control-string :format-arguments arguments)) (defun expected (expected &key got) (fail "expected ~A, got ~A instead" expected got)) Here's an example of how I use EXPECTED/FAIL: (deftest dec.ascii.2 () (handler-case (octets-to-string (ub8v 97 128 99) :encoding :ascii :errorp t) (character-decoding-error (c) (is (equalp #(128) (character-decoding-error-octets c))) (is (eql 1 (character-coding-error-position c))) (is (eq :ascii (character-coding-error-encoding c)))) (:no-error (result) (expected 'character-decoding-error :got result)))) Another idea would be to use SIGNALS: (let ((c (signals (octets-to-string ...)))) ;; test C ) However, it seems to me that this would introduce bogus failures and/or errors when SIGNALS returns NIL that would make debugging more difficult. Any suggestions? Since babel-tests was using RT, it had lots of small tests like this: (deftest foo form retval1 retval2 ...) so I found myself doing a lot of (deftest foo () (is (equalp form retval1))) and worse for multiple return values. Before writing the RETURNS macro I wrote a DEFSTEST macro that works like RT's: (defmacro defstest (name form &body return-values) "Similar to RT's DEFTEST." `(deftest ,name () (returns ,form ,@(mapcar (lambda (x) `',x) return-values)))) I like it because I'm used to RT and it made porting most of these small tests a simple matter of replacing DEFTEST with DEFSTEST but I'm not sure how useful it'd be in general. There a few itches I'd like to scratch before porting other test suites I maintain. Here's my TODO list, let me know if you have any objections or better alternatives to these features: * make FUNCALL-TEST-WITH-FEEDBACK-MESSAGE print a list of failed tests (perhaps along with descriptions of the failed assertions). This could be useful for bug reports, buildbots, and other non-interactive use cases. * implement a functional interface for marking expected failures. E.g.: #+lispworks (expected-failure 'some-test-lispworks-does-not-like). * implement a way of running tests through EVAL, instead of compiling. We use this in cffi-tests to test whether removing compiler-macros introduces bugs and stuff like that. * minor thing: export STEFIL::ROOT-SUITE so that one can write (deftest (foo :in root-suite)). Finally, reducing the dependencies to external libraries would help with lobbying for porting cffi-tests (and others) to Stefil. Would that be in any way acceptable for you guys? Thanks for Stefil and sorry for the long email. -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/

1 0