Daniel Herring wrote:
On Mon, 5 Oct 2009, Robert Goldman wrote:
Daniel Herring wrote:
On Mon, 5 Oct 2009, Robert Goldman wrote:
Gary King wrote:
Hi Robert,
I don't believe that this is a general solution, for two reasons:
I agree that it isn't a general solution especially since there is no interface/API for clients to do anything with an ASDF operation! It might, however, be a small step in the right direction.
An alternative solution would be to provide a :stream or :filename init argument for the test-op operation class and bind a dynamic variable around every perform, making the stream or filename available for writing....
Why serialize the data? Could we design a structured API to be used by other tools?
...
Of course ASDF doesn't need to reinvent testing; there are plenty of existing frameworks to choose from. http://www.cliki.net/test_framework
What I am after is an ASDF test-op that will adapt as well as possible to the widest possible set of testing frameworks. I agree that we should not attempt to reinvent testing.
That is why I have been suggesting that we provide a test operation that binds a stream --- because most of the test frameworks I have worked with provide a test report, rather than returning results.
I don't believe that having the test-op return a result will work, given the current ASDF execution model.
...
Given that, how about deprecating/removing test-op? Then people can continue to simply load sysname-test.asd and run tests however they want.
I see value in creating a widely accepted testing framework. I don't see the benefit in simply binding a stream variable.
Let me clarify. I /do/ see the value of providing a test-op.
We have our own testing framework at my company, NST (see presentation hosted at tc-lispers.org for more information). We have provided a back-end for it that hooks into ASDF, so that people can do
(test-system <foo>)
and our NST tests will run and a report will be printed.
This isn't fully ideal, though, since the results of tests will be mingled in the console output together with compilation results, which makes finding the test outputs a little more difficult than it should be. So I think providing a standardized output stream would be helpful, and probably generally helpful, since no matter what your unit-testing library, if it writes a report to the console, and is invoked from ASDF, it's likely to suffer from the same problem of output interleaving.
I don't think that ASDF needs to provide a test framework; all it needs to do is to provide hooks for invoking existing test framework(s).
Best, r