Tobias C. Rittweiler wrote:
Robert Goldman writes:
Juan Jose Garcia-Ripoll wrote:
You are just imposing too much complexity. If I want to test package Cl-UNICODE, I do nont want to test FLEXI-STREAMS or U-SOCKETS. Tests should be atomic and not generate a tree of actions like ASDF does not for everything.
Please see earlier discussion about this topic.
The point I made there is that systems may have COMPONENT subsystems such that you want to test the entire system together.
Consider, e.g., a DB library with multiple backends, each described in a separate system. In order to do the test-op on the DB library you want to do the test on all active backends. Similarly, McCLIM has multiple graphics display backends, not all of which are loaded at a given time. I work actively on three (or four, depending on how you count) large CL-based applications. Each one of them is made up of multiple ASDF subsystems.
Yes, that is a valid use case. No one disputes that.
What if we start with the simple case of only carring one bit of meta-information (successp), merging is nothing more than ORing together the results.
This can be done in a way to satiesfy current needs without restricting future improvement (which then can be based on actual experience in the field.)
Actually, I don't think that the hard part is deciding how to merge things together. I agree with you that the merge operation can be made a generic function, solving this problem.
The greater problem, I believe, is determining /what/ operations to merge together.
Consider the case of a system, X, on which I invoke the TEST-OP. Doing so can create a plan (through TRAVERSE) which contains an arbitrary number of operations necessary to perform TEST-OP on X, including potentially many LOAD-OPs and COMPILE-OPs. How many of these operations need to be inspected for success or failure?
One might think "all and only the test-op"s, but I can say from sad experience that this is not so --- a large program on which I am now working came to a halt for several hours because it did TEST-OP, and the TEST-OP didn't yield a report because some subsystem failed to build. We spent quite a while looking for the test-op's output which we, of course, did not find. So certainly, one should make sure that the test-op handle this properly. For interactive use, this is not a problem --- if the load fails, an exception will be raised. But for NON-interactive use (in our case on a build-and-test server) it's not so obvious.... A build-and-test server is very handy on medium to large size projects to verify that someone committing a patch is not breaking the build.
As I said, perhaps the ancestor links among operations may give a solution to this problem: I simply haven't had time to investigate thoroughly.
I will try to get some time in the near future to discuss this matter with my colleague, John Maraist, who has done the most recent work on integrating the NST unit test library and ASDF. I will report back anything useful I learn.
best, Robert