Juan Jose Garcia-Ripoll writes:
On Mon, Oct 19, 2009 at 12:00 AM, Robert Goldman wrote:
[for the record; the topmost paragraph came from me:]
In particular because it's my impression that the problems have been over-stated.
I think so. And this is indicated by the following paragraphs
- Not all regression test frameworks are functional, returning values.
Some write reports instead.
That does not mean anything. ASDF may impose one behavior w.r.t to the output and the package maintainer may decide to output T saying "hey, there might be errors, but I do not know." Later on other software and test suites will catch up. With RT, for instance it is trivial to grab the number of failed tests.
- One needs to come up with a means of combining operation results
that takes into account the structure of the plan that traverse produces and that operate then executes. If in order to test-op X, I must test-op A and B, how do I combine together the test results from A, B, and X into a top-level operation result. Perhaps OPERATION-ANCESTOR can be pressed into service.
You are just imposing too much complexity. If I want to test package Cl-UNICODE, I do nont want to test FLEXI-STREAMS or U-SOCKETS. Tests should be atomic and not generate a tree of actions like ASDF does not for everything.
As an interim solution, there can be some merge operation which, for the moment, just bails out if multiple results really have to be combined.
- Is there some way to do this such that the regression-testing
framework need not be made aware of ASDF?
No, there is none. Regression tests are libraries which are used by people using ASDF. Eventually they may provide their own operations, extending what ASDF provides, but currently it is better to have some bare minimum from ASDF and let each package maintainer decide how they compute the success of their test suite.
Right.
In particular: between ASDF and the regression test framework always stands a _human being_, the maintainer of the software package. It's his duty to interconnect his choice of third-party test framework with ASDF -- that is if he really wants to. If he doesn't, well, he just shouldn't.
Existing practise does luckily not matter much in this case. It's about meta-information that nobody can depend on at the moment because it does not exist. And if it exists, it'll matter to people who care and will do the right thing. (Which may include fixing other people's code; but see that it's in _their_ own interests to do so, so they will.)
Blocking this development just because there are 5 test suites and you do not know how to combine them with ASDF is really absurd. ASDF's specifications can not depend on what your company or other people's companies are setting up for their workflow.
Blocking is way too strong a word. After all, both of us want to piggy-back onto work of others. We can just hope to make our case clear to those who contribute code---or contribute ourselves.
-T.