Congratulations on getting the divergent ANSI-TEST suite codebases back together in one place, as well as establishing this mailing list. As a Common Lisp implementer who only really started studying the language a decade ago, I have profited immeasurably from figuring out why ABCL fails a given ANSI-TEST while other implementations don't.
I would like to point out a few "extensions" to the ANSI-TEST suite that ABCL has added over the years so that other implementations might profit from them and/or that some of the ideas might be included in the ANSI-TEST suite itself. Upon reviewing the implementation of these ideas, parts of me cringe at how my younger self implemented things so I wouldn't necessarily recommend adoption of our code wholesale.
1) ASDF
By the nature of how ANSI-TEST loads and runs itself, encapsulation as a "true" ASDF system (i.e. one where all the source is declared) seems a little farfetched. And maybe it is useful to be able to run the suite in places where ASDF is not available, such as when bootstrapping a new implementation. Nevertheless, we put some work into encapsulating the ANSI-SUITE into ASDF to the point that it can be loaded and the tests run that might be of interest.
[ABCL defines three ASDF systems][1], namely ANSI-RT, ANSI-COMPILED, and ANSI-INTERPRETED. ANSI-RT encapsulates the regression test framework, while ANSI-COMPILED and ANSI-INTERPETED provide definitions to the point that ASDF:TEST-SYSTEM can run the compiled or interpreted tests respectively. This helps with automation of the tests in ABCL's build mechanism. The one caveat is that one cannot run tests in "parallel" on, say, different versions of your implementation that you are regression testing due to the way that ANSI-TEST uses the filesystem of the ANSI-TEST itself to keep fasls and record state for tests in the course of running the suite.
[1]: http://abcl.org/trac/browser/trunk/abcl/abcl.asd#L76
2) ABCL.TEST.ANSI
In order to support the necessary shenanigans to make the ANSI-COMPILED and ANSI-INTERPRETED definitions work, we [created a small set of utility routines][2]. CLEAN-TESTS performs the removal of intermediate files that the old Makefile used to do. DO-TESTS-MATCHING will run only those tests whose symbols match a regular expression, which is useful for repeatedly running a subset of tests.
[2] http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/abcl-ansi.lisp
3) ANSI-TEST error "database"
We [created a simple s-expr "database"][4] for recording test failures and then [reporting on the results][3]. ABCL is a unique CL implementation because it is hosted on a JVM for which there are multiple implementations and whose behavior often varies by operating system so running the ANSI-TEST suite has proven to be fairly sensitive to hosting JVM implementation version and operating system. Thus, such a database of regressions has proven to be of special importance to ABCL. That being said, my implementation of this idea is pretty amateurish in hindsight, so I think what is important here is the idea rather than the implementation.
[3]: http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/parse-ansi-errors.lis... [4]: http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/ansi-test-failures