Gary King wrote:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
FWIW, I recently changed operate-on-system (and therefore asdf:test-system, asdf:load-system, etc.) to return the operation instance that they processed.
If ASDF also added and documented (shocking!) a few canonical slots in the test-op class then these could be used to transmit information. As a first pass, I'd suggest
test-properties - a property list of test system specific stuff test-result - a keyword from the list :success, :failures,:errors, errors-and-failures
Some time ago, I proposed that ASDF provide a stream argument to the test-op, providing a place into which the testing tool could dump its test report for human inspection. I can't say that this suggestion received universal approbation. Or disapprobation. It merely received near-universal lack of interest! ;-)
Sigh. I think this is a good idea; if we added a test-stream slot to the test-op, then test systems could find it there.
A tricky part is that test systems may not want to rely on the existence of ASDF while running. So they'll need some sort of adaptor that adds stuff to the ASDF operation if it exists. Would it make sense to define and export *current-operation* to make this easier?
You'd still need the asdf stuff to get at this. The alternative would be to define a special variable that gives direct access to the stream, which is not an asdf-dependent entity.
The problem with that approach is that you're pretty much doomed to crap up the CL-USER namespace.
Related to the question of what the test-op should provide to its invoker is the question of how dependencies should be propagated.
I think that the default is that test-op depends on load-op.
Right, but what about testing system <foo> --- do we test <foo>'s depended on systems? My guess is that the correct answer to this is "it depends." If you have a standard set of libraries, the answer will typically be "no." OTOH, if you have a very large application (e.g., a web site with many components providing different features), you may well want to test the subsystems that are part of the top-level system.
other thoughts?
We have found in practice that we end up creating a test-<foo> system definition to complement our <foo> system definitions. The reason for this is that the existing ASDF structures have only components that are unconditional. For the test op we typically want to load new files. A second item is that the test system typically :depends-on the test framework library.
This suggests a question: is there a nice way to specify additional files that need to be loaded ONLY for the test-op? I can think of some ways to achieve this with the existing ASDF framework, but they would be cumbersome and non-standard. Would it be possible to simply add a :test-components argument to defsystem?
Note that the use of additional <test>-foo systems also helps us tailor subcomponent testing.
Another note --- I found in an earlier attempt to tweak the test-op that it worked poorly to subclass operations (as opposed to components). IIRC, this was because of the way dependent operation instances are built. Alas, I didn't take good notes when I did this, and have garbage-collected much of what I learned.
r