Faré wrote:
Maybe ASDF is the wrong place to try to standardize testing infrastructure?
This is the conclusion I have reached, as well. I was hoping that some very weak standard could be arrived at that would make the test-op more generally useful to people installing systems, so that they could simply run the test-op and easily tell whether or not the test operation was successful.
However, it may be that it's just a combination of features of our test framework and the way we have built our systems that makes it difficult for us (the original problem --- we find that we get compilation and loading output mixed together with the test output), so this is not a place where the ASDF community can readily get consensus.
So please consider the suggest dropped.
I may try to come back to the list with a proposal for modifying the MAKE-SUB-OPERATION function so that it makes specializing operations (including the test-op) easier. That would be a more general proposal that might make many things easier for ASDF system specifiers.
I mean, maybe instead the authors of various test infrastructures should have a common list where they discuss interoperability, reporting, and a declarative way of specifying dependencies between test suites, between files and test suites, etc.?
Since for instance XCVB doesn't have testing support yet, but testing support is amongst our eventual goals, I would be eager to have feedback on how a new build system could "do things right" in this regard -- or perhaps avoid doing anything, and just plugging into something else.
Let me suggest a few desiderata:
1. Make it possible for operation performance to return values. Think about what the types of these values should be while designing this facility.
2. Provide a standard means for specifying how sub-operation values should roll up to the values of their parents.
3. If you are taking the plan then act design framework, think about how to keep the (direct) child operations of a parent in some kind of procedure call envelope (dynamic scope extent).
4. Deal with the difference between how different operations should handle values and failure. E.g., a test operation should simply accumulate failures. A build operation should almost always fail at the first point of failure.
Hope that helps,
r