[fiveam-devel] Unit Test Framework
Faithful hackers, I decided to take up the challenge laid down here http://fare.livejournal.com/169346.html and try to consolidate the Common Lisp unit testing frameworks. I have written a framework that aims to consolidate all the major features of all your frameworks mentioned in this blog http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.... . You can find it on Github https://github.com/tgutu/clunit. I also wrote a blog on the development of the framework and reasons for it here http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/ if you are interested. I would very much appreciate it, if you could join me in this effort and we all work together torwards making this project a success. Regards, Tapiwa
On a quick skim, looks like nice work. If you want to see if it handles a lot of the cases (better or as well) that lisp-unit was designed for, see http://www.cs.northwestern.edu/academics/courses/325/programs/exercise-tests... This is what tests my students' solutions to exercises from Graham and elsewhere. I'd prefer to use a standard test framework in my class, but I would want that particular file to continue to be simple for novices to read and understand, since they're the target audience. I'll try making the translation myself sometime but it won't be for the next few months at least, due to time constraints. I notice that you didn't include an "assert-prints" though obviously it's not hard to use some wrapper macro like (assert-equal "...." (collect-output (print-dots 4))) I would recommend adding something comparable to lisp-unit's assert-equality that lets you handle special equality testers, like set-equal, epsilon-equal, etc. (I used to call it assert-predicate but that implied a unary rather than binary predicate.) Nice job. On Nov 10, 2012, at 11:41 AM, Tapiwa Gutu wrote: Faithful hackers, I decided to take up the challenge laid down here http://fare.livejournal.com/169346.html and try to consolidate the Common Lisp unit testing frameworks. I have written a framework that aims to consolidate all the major features of all your frameworks mentioned in this blog http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks..... You can find it on Github https://github.com/tgutu/clunit. I also wrote a blog on the development of the framework and reasons for it here http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/ if you are interested. I would very much appreciate it, if you could join me in this effort and we all work together torwards making this project a success. Regards, Tapiwa Christopher Riesbeck Home page: http://www.cs.northwestern.edu/~riesbeck Calendar: http://calendar.yahoo.com/criesbeck
Hey Chris, Thanks for the quick response. I took a look at the link you posted. The syntax I used was most heavily influenced by that in lisp-unit because what any given function or macro did was very clear from the name without having to read the documentation first. Names like ASSERT-TRUE and ASSERT-EQUAL are pretty self explanatory! Also the syntax is very simple and easy to read and use. Because the syntax is so close to that of lisp-unit I will make a quick translation with a few changes and test it. The changes would be as shown below, the rest is pretty much the same. (in-package :cs325-user) ;;; Define a suite for all our tests since CLUnit only groups according to test suites instead of the implicit package grouping in lisp-unit. (defsuite cs325-user ()) (define-test testname . body) -> (deftest testname (cs325-user) . body) I had left out ASSERT-PRINTS simply because I had not considered its uses well enough. But after you pointed it out, I realised having a built in form to test the messages you print in your library would be very handy. The definition of ASSERT-TRUE that I used is actually a predicate tester according to its common use in Lisp conditional forms such as IF and WHEN. I figured if you want to test for the specific value T you would use (assert-eq t form). (assert-equality 'set-equal '(car) (functions-referenced '(car l))) -> (assert-true (set-equal '(car) (functions-referenced '(car l))) There is an idea I have been toying with in my head. It seems the library generally works now so I think its now safe to write unit tests for the library itself. Now, I obviously need test the lower level functions and make sure that they are doing the correct thing. So I have a situation were I want to test A and B, but since B depends of A, if A fails its test there is no point testing B. I would need to first fix A before I can trust what B will do. Basically, I want to be able to short circuit tests. I want to add this without changing the current behaviour. Since execution of tests in a test suite is unordered, you would place the dependent tests in a child test suite. So something like this: 1. (defsuite A ()) 2. (defsuite B (A)) 3. Define low level function tests in A. 4. Define tests in B that depend on functions tested in A. 5. (run-suite 'A :stop-on-fail t) The RUN-SUITE keyword argument :use-debugger, is for interactive tests, this option would work for short circuiting batch tests. What do you think? On 10 November 2012 21:28, Christopher K Riesbeck < c-riesbeck@northwestern.edu> wrote:
On a quick skim, looks like nice work. If you want to see if it handles a lot of the cases (better or as well) that lisp-unit was designed for, see
http://www.cs.northwestern.edu/academics/courses/325/programs/exercise-tests...
This is what tests my students' solutions to exercises from Graham and elsewhere. I'd prefer to use a standard test framework in my class, but I would want that particular file to continue to be simple for novices to read and understand, since they're the target audience.
I'll try making the translation myself sometime but it won't be for the next few months at least, due to time constraints.
I notice that you didn't include an "assert-prints" though obviously it's not hard to use some wrapper macro like
(assert-equal "...." (collect-output (print-dots 4)))
I would recommend adding something comparable to lisp-unit's assert-equality that lets you handle special equality testers, like set-equal, epsilon-equal, etc. (I used to call it assert-predicate but that implied a unary rather than binary predicate.)
Nice job.
On Nov 10, 2012, at 11:41 AM, Tapiwa Gutu wrote:
Faithful hackers,
I decided to take up the challenge laid down here http://fare.livejournal.com/169346.html and try to consolidate the Common Lisp unit testing frameworks. I have written a framework that aims to consolidate all the major features of all your frameworks mentioned in this blog http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.... .
You can find it on Github https://github.com/tgutu/clunit. I also wrote a blog on the development of the framework and reasons for it here http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/if you are interested.
I would very much appreciate it, if you could join me in this effort and we all work together torwards making this project a success.
Regards, Tapiwa
Christopher Riesbeck Home page: http://www.cs.northwestern.edu/~riesbeck Calendar: http://calendar.yahoo.com/criesbeck
On 11/10/12 Nov 10 -11:41 AM, Tapiwa Gutu wrote:
Faithful hackers,
I decided to take up the challenge laid down here http://fare.livejournal.com/169346.html and try to consolidate the Common Lisp unit testing frameworks.I have written a framework that aims to consolidate all the major features of all your frameworks mentioned in this blog http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.....
You can find it on Github https://github.com/tgutu/clunit. I also wrote a blog on the development of the framework and reasons for it here http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/ if you are interested.
I would very much appreciate it, if you could join me in this effort and we all work together torwards making this project a success.
Executive summary: Hey! Not so fast! Longer: I'm in favor of Faré's approach, but I feel like you are jumping in faster than I'd like to see us go. Faré's article suggests these steps: 1. Pick your favorite problem domain 2. Identify all libraries that address it 3. Work with various library authors 4. Pick the most promising library 5. Declare it THE library It feels to me like you are skipping steps 3 and 4. I guess I'd like to see a wider discussion of this issue before jumping to #5. At SIFT, we ended up choosing FiveAM as THE library, for our purposes. Are you saying that cl-unit is the most promising? If so, why? Or are you saying that "None of the above" is the most promising, and we need to build YA framework from scratch? That is not at all what Faré was arguing for, as I read it. Certainly the write-up by Olin Shivers should be read as a push to fix an existing framework, rather than develop a new one. If you are proposing that we take cl-unit, what are you planning to do to provide a compatibility layer for people whose decisions are different for yours? Are you going to make it possible for people who use FiveAM to migrate their existing test suites without a lot of work? What's your plan to do this task that Faré identifies: "To fully solve the problem domain, we must identify all the libraries, and all their current and desired features, so that we may provide a 100% replacement to each and every one of these existing libraries, based on an architecture that makes all the future features possible." That sounds to me like the right thing to do now is try to come to consensus around a requirements document. I don't think that jumping into setting up a repo and slinging code is the right place to be now. Please do a little more consensus building and more "work with various library authors" and working with library users, so that you are not building YA 80% solution. Thanks! r
Tapiwa Gutu <tapiwa.gutu@gmail.com> writes:
Faithful hackers,
I decided to take up the challenge laid down here http://fare.livejournal.com/169346.html and try to consolidate the Common Lisp unit testing frameworks. I have written a framework that aims to consolidate all the major features of all your frameworks mentioned in this blog http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.....
You can find it on Github https://github.com/tgutu/clunit. I also wrote a blog on the development of the framework and reasons for it here http://ml.sun.ac.za/2012/11/09/ developing-a-unit-test-framework-part-1/ if you are interested.
i mean no disrespect to your effort, but i think it'd be better to improve one of the existing test frameworks (and i'm partial to fiveam) than to try and start from scratch. now, if fiveam really doesn't (or can't) do what you want/need it to do, then i'd love to talk about what that is. if you end up writing your own test framework, well, i can totally understand that too. i also think that this problem isn't nearly as bad as it seems, fiveam already does many of the things the other test suites do, but the documentation is pretty horrible (so nobody knows this). did you know about 5am:debug! ? (i didn't. until a few days ago). so, here's an attempt at improving fiveam's documentation (this is 60 or 70% complete): http://bese.it/fiveam/manual.html i'd love someone to have a quick look at it and tell me if it makes sense (i know fiveam too well to be a good judge) or if it needs to be fundamentally changed somehow. it all lives in this repo: https://github.com/segv/fiveam which, other than the docs contains a few minor cleanups and one new feature (no pull request yet). ps - you know what would be really cool? source location of the failing test (if i have 20 is checks in a test and one of them fails it'd be nice to know the filename and line number of those checks which failed. i'd settle form something that was #+(or sbcl ccl)...) pps - to get stefil's debugger when a test fails in fiveam, just do (setf 5am:*debug-on-failure* t), if you want low over head just ignore suites (the default suite, nil, always exists) and use (5am:run!) to run all the currently defined tests, if you want ever lower overhead then set 5am:*run-test-when-defined* and you can skip the call to run!. -- -marco
On 11/30/12 Nov 30 -12:42 PM, Marco Baringer wrote:
ps - you know what would be really cool? source location of the failing test (if i have 20 is checks in a test and one of them fails it'd be nice to know the filename and line number of those checks which failed. i'd settle form something that was #+(or sbcl ccl)...)
I am familiar with how to set this up for ACL. If someone will create a framework for doing this (e.g., figure out how we decide which test to look for if the same name is used in two different suites, etc.), then I would be happy to contribute code for ACL. I suspect that a lot of this will be implementation-specific. We can all support our favorites to get a generally useful implementation.
participants (4)
-
Christopher K Riesbeck -
Marco Baringer -
Robert Goldman -
Tapiwa Gutu