Congratulations on getting the divergent ANSI-TEST suite codebases back together in one place, as well as establishing this mailing list. As a Common Lisp implementer who only really started studying the language a decade ago, I have profited immeasurably from figuring out why ABCL fails a given ANSI-TEST while other implementations don't.
I would like to point out a few "extensions" to the ANSI-TEST suite that ABCL has added over the years so that other implementations might profit from them and/or that some of the ideas might be included in the ANSI-TEST suite itself. Upon reviewing the implementation of these ideas, parts of me cringe at how my younger self implemented things so I wouldn't necessarily recommend adoption of our code wholesale.
1) ASDF
By the nature of how ANSI-TEST loads and runs itself, encapsulation as a "true" ASDF system (i.e. one where all the source is declared) seems a little farfetched. And maybe it is useful to be able to run the suite in places where ASDF is not available, such as when bootstrapping a new implementation. Nevertheless, we put some work into encapsulating the ANSI-SUITE into ASDF to the point that it can be loaded and the tests run that might be of interest.
[ABCL defines three ASDF systems][1], namely ANSI-RT, ANSI-COMPILED, and ANSI-INTERPRETED. ANSI-RT encapsulates the regression test framework, while ANSI-COMPILED and ANSI-INTERPETED provide definitions to the point that ASDF:TEST-SYSTEM can run the compiled or interpreted tests respectively. This helps with automation of the tests in ABCL's build mechanism. The one caveat is that one cannot run tests in "parallel" on, say, different versions of your implementation that you are regression testing due to the way that ANSI-TEST uses the filesystem of the ANSI-TEST itself to keep fasls and record state for tests in the course of running the suite.
[1]: http://abcl.org/trac/browser/trunk/abcl/abcl.asd#L76
2) ABCL.TEST.ANSI
In order to support the necessary shenanigans to make the ANSI-COMPILED and ANSI-INTERPRETED definitions work, we [created a small set of utility routines][2]. CLEAN-TESTS performs the removal of intermediate files that the old Makefile used to do. DO-TESTS-MATCHING will run only those tests whose symbols match a regular expression, which is useful for repeatedly running a subset of tests.
[2] http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/abcl-ansi.lisp
3) ANSI-TEST error "database"
We [created a simple s-expr "database"][4] for recording test failures and then [reporting on the results][3]. ABCL is a unique CL implementation because it is hosted on a JVM for which there are multiple implementations and whose behavior often varies by operating system so running the ANSI-TEST suite has proven to be fairly sensitive to hosting JVM implementation version and operating system. Thus, such a database of regressions has proven to be of special importance to ABCL. That being said, my implementation of this idea is pretty amateurish in hindsight, so I think what is important here is the idea rather than the implementation.
[3]: http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/parse-ansi-errors.lis... [4]: http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/ansi-test-failures
"Mark" == Mark Evenson evenson@panix.com writes:
[snip]
Mark> 1) ASDF
Mark> By the nature of how ANSI-TEST loads and runs itself, encapsulation as a Mark> "true" ASDF system (i.e. one where all the source is declared) seems a Mark> little farfetched. And maybe it is useful to be able to run the suite Mark> in places where ASDF is not available, such as when bootstrapping a new Mark> implementation. Nevertheless, we put some work into encapsulating the Mark> ANSI-SUITE into ASDF to the point that it can be loaded and the tests Mark> run that might be of interest.
This is interesting. It sounds nicer than running a makefile where I always have to look to see what the magic variables are to run what I want.
Mark> 2) ABCL.TEST.ANSI
Mark> In order to support the necessary shenanigans to make the ANSI-COMPILED Mark> and ANSI-INTERPRETED definitions work, we [created a small set of Mark> utility routines][2]. CLEAN-TESTS performs the removal of intermediate Mark> files that the old Makefile used to do. DO-TESTS-MATCHING will run only Mark> those tests whose symbols match a regular expression, which is useful Mark> for repeatedly running a subset of tests.
This also sounds interesting because I often do want to run just a subset.
Mark> 3) ANSI-TEST error "database"
Mark> We [created a simple s-expr "database"][4] for recording test failures Mark> and then [reporting on the results][3]. ABCL is a unique CL Mark> implementation because it is hosted on a JVM for which there are Mark> multiple implementations and whose behavior often varies by operating Mark> system so running the ANSI-TEST suite has proven to be fairly sensitive Mark> to hosting JVM implementation version and operating system. Thus, such Mark> a database of regressions has proven to be of special importance to Mark> ABCL. That being said, my implementation of this idea is pretty Mark> amateurish in hindsight, so I think what is important here is the idea Mark> rather than the implementation.
Again, this sounds pretty cool too. Cmucl has a few extensions that cause the test suite to fail. It would be nice to annotate these test as expected failures. Idealy, a reason for the failure would be part of the annotation. I don't run the test suite that often anymore, but when I do, I always forget exactly which tests are known expected failures and which are new. Having some kind of annotation (other than comments on the test itself) would be awesome!
-- Ray
armedbear-devel@common-lisp.net
-
Mark Evenson -
Raymond Toy