On Sep 25, 2019, at 21:06, Vladimir Sedach vas@oneofus.la wrote:
Mark Evenson evenson@panix.com writes:
Each flavor of testing framework needs to write a simple adaptor that returns a condition containing a boolean indicating success or failure and the testing framework specific results.
Unfortunately, it has the same downsides. There is only a subset of the test libraries supported. The code depends on the internal workings of the test libraries and is often out of date. The projects cannot be used as-is to test arbitrary systems (with cl-test-grid, you have to manually add configuration for each system to testsuites/testsuites.lisp, asdf-test-harness seems to depend on emotiq configuration files).
This approach has not worked well in the past 8 years (cl-test-grid was started in 2011), and I do not think it is viable.
What you have also done in asdf-test-harness, adding an ASDF-TEST-SYSTEM-FAILURE condition to ASDF, is a better approach. I have shown how this can be used by test libraries (with working implementations for FiveAM³ and rove⁴). This removes the need for asdf-test-harness, cl-test-grid, etc. to each have to implement custom adapters for every different test library, completely eliminating the dependency between test harnesses and particular test libraries. The test libraries themselves do not need any new dependencies either.
This decouples the systems and will make projects like asdf-test-harness much easier to extend and maintain.
Thanks for taking the time to understand my code enough to cogently critique the two approaches.
As I understand it, both approaches share the need to add a condition to ASDF that indicates that an invocation of an ASDF test-op has failed.
The difference is that you propose getting changes to all the various test framework to signal this error condition with a subclass that contains meaningful information for that given framework. Whereas, I (and cl-test-grid)--who was in a hurry to get something working without being able to wait for modifications to the test suites--needs to write an adaptor for each test suite. And consequently these adaptors will be often be “brittle” depending on implementation details that may not be constant over time to work.
If this is indeed an accurate summary, then I would endorse your patch to ASDF as necessary for both approaches with the following suggestions for improvements:
1. Have a slot in your base condition class TEST-OP-TEST-FAILURE in which one can record the ASDF component which caused the failure. It is probably possible to dig this information out of the stack, but that will be messy. This would also allow for distinguishing when multiple TEST-OP-TEST-FAILURES are signaled from a single ASDF:TEST-OP invocation, as will be the case when one “chains” test invocation over many ASDF systems.
2. Provide an implementation of the subclass of TEST-OP-TEST-FAILURE that contains the basic structure of a reporter class for the information that should be present in all test frameworks, namely the total number of tests run, the number of failed tests, the identities of the failed tests, and a slot for a human readable error message, along with a reporter function that displays this information. Having an implementation class to work from would make it easier for test frameworks to adapt.
3. Go ahead and define the subclass of this condition when no tests have been run.
4. As for adoption by test framework your strategy, we will have the problem that a given test framework won’t want to adopt the conditions because it isn’t in the version of ASDF they are using, or can easily get a hold of. To solve this, we might somehow define the code within the ASDF source tree so that one can make a standalone ASDF system (“ASDF-TEST-CONDITIONS” or some such) that one may include separately from actually upgrading ASDF.
Sincerely, Mark Evenson