Mark Evenson evenson@panix.com writes:
Each flavor of testing framework needs to write a simple adaptor that returns a condition containing a boolean indicating success or failure and the testing framework specific results.
I am not sure if you are aware of cl-test-grid¹ - it takes the same approach.
Unfortunately, it has the same downsides. There is only a subset of the test libraries supported. The code depends on the internal workings of the test libraries and is often out of date. The projects cannot be used as-is to test arbitrary systems (with cl-test-grid, you have to manually add configuration for each system to testsuites/testsuites.lisp, asdf-test-harness seems to depend on emotiq configuration files).
This approach has not worked well in the past 8 years (cl-test-grid was started in 2011), and I do not think it is viable.
What you have also done in asdf-test-harness, adding an ASDF-TEST-SYSTEM-FAILURE condition to ASDF, is a better approach. I have shown how this can be used by test libraries (with working implementations for FiveAM³ and rove⁴). This removes the need for asdf-test-harness, cl-test-grid, etc. to each have to implement custom adapters for every different test library, completely eliminating the dependency between test harnesses and particular test libraries. The test libraries themselves do not need any new dependencies either.
This decouples the systems and will make projects like asdf-test-harness much easier to extend and maintain.
As an example, writing a script to test systems in different implementations now becomes trivial.⁵
¹ https://github.com/cl-test-grid/cl-test-grid ² https://github.com/easye/asdf-test-harness/blob/master/asdf-test-harness.lis... ³ https://github.com/sionescu/fiveam/pull/58 ⁴ https://github.com/vsedach/rove/commit/f6f8822eedc61d131c3b1d37b45c6d48cefcf... ⁵ https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-te...
-- Vladimir Sedach Software engineering services in Los Angeles https://oneofus.la