Daniel Herring wrote:
On Mon, 5 Oct 2009, Robert Goldman wrote:
Gary King wrote:
Hi Robert,
I don't believe that this is a general solution, for two reasons:
I agree that it isn't a general solution especially since there is no interface/API for clients to do anything with an ASDF operation! It might, however, be a small step in the right direction.
An alternative solution would be to provide a :stream or :filename init argument for the test-op operation class and bind a dynamic variable around every perform, making the stream or filename available for writing....
Why serialize the data? Could we design a structured API to be used by other tools?
What if each test logged messages to asdf:*test-stream* and finished by calling (asdf:test-result test-description-string result-keyword)
DejaGnu has a good description of test results; the keywords from good to bad could be :xpass, :xfail, :untested, :unresolved, :unsupported, :pass, and :fail. http://www.gnu.org/software/dejagnu/manual/x47.html
Then asdf:test-op could return (values worst-result results-list).
Of course ASDF doesn't need to reinvent testing; there are plenty of existing frameworks to choose from. http://www.cliki.net/test_framework
What I am after is an ASDF test-op that will adapt as well as possible to the widest possible set of testing frameworks. I agree that we should not attempt to reinvent testing.
That is why I have been suggesting that we provide a test operation that binds a stream --- because most of the test frameworks I have worked with provide a test report, rather than returning results.
I don't believe that having the test-op return a result will work, given the current ASDF execution model. If we are testing system X that has A and B dependencies, then what will happen will be something like
.... testing A's components... test-op A....testing B's components ... test-op B .... test other X components ... test-op X
It's not obvious to me how we take the results from test-op A and test-op B and roll them up into test-op X. Nor do I see how, in general, a test-op done in LIFT for A, FiveAM for B and NST for X, can be made to communicate with each other.
It's not ideal, but I think a separate stream may be about as good as we can do.
Best, r