Tobias C. Rittweiler wrote:
Robert Goldman writes:
Faré wrote:
Maybe ASDF is the wrong place to try to standardize testing infrastructure?
This is the conclusion I have reached, as well. I was hoping that some very weak standard could be arrived at that would make the test-op more generally useful to people installing systems, so that they could simply run the test-op and easily tell whether or not the test operation was successful.
However, it may be that it's just a combination of features of our test framework and the way we have built our systems that makes it difficult for us (the original problem --- we find that we get compilation and loading output mixed together with the test output), so this is not a place where the ASDF community can readily get consensus.
So please consider the suggest dropped.
Please excuse me that I cannot grow at peace with this conclusion.
The missed utility is far too great to let this rest:
Juanjo's automatic regression testing work to check for regressions in ECL is dead on, and I'm sure other implementators would be interested in such infrastructure as well.
The same is true for D Herring and his LibCL project which looks very promising, but is also a candidate for automatic regression testing. I think that Daniel is the right guy for the job and I hope he'll have the necessary perseverance for what he has started.
I have to excuse myself again because I'm not familiar with ASDF in any great detail; on the other hand, ASDF -- unlike most other Lisp project -- consists of over half a dozen of people who do know it pretty well, so I cannot believe that we can not come up with some solution.
In particular because it's my impression that the problems have been over-stated.
I think there are some problems which are real, but perhaps not insuperable:
1. Not all regression test frameworks are functional, returning values. Some write reports instead.
2. One needs to come up with a means of combining operation results that takes into account the structure of the plan that traverse produces and that operate then executes. If in order to test-op X, I must test-op A and B, how do I combine together the test results from A, B, and X into a top-level operation result. Perhaps OPERATION-ANCESTOR can be pressed into service.
3. Is there some way to do this such that the regression-testing framework need not be made aware of ASDF? Ideally, ASDF and the regression-testing framework would be defined independently, and some additional combination code would be defined that would link the two together, and that would be loaded downstream. I'd be reluctant to see a solution that demanded our regression-testing frameworks become dependent on ASDF. I believe that this problem could be easily circumvented by defining new systems like ASDF-FIVEAM, ASDF-NST, ASDF-STEFIL. At my company, we have already been working on ASDF-NST, but it has not reached a fully satisfactory state.
best, r