Gary King wrote:

Hi Robert,

Ignoring any thinking about your actual questions <smile>...

...

This operation should test the specified component. ASDF does not (currently) provide a method for returning a value from the test-op (or any other operation),

This is no longer true; ADSF does return the operation so if there was also a way for test systems to annotate the operation, some things would become well.

I don't believe that this is a general solution, for two reasons: 1. The test frameworks aren't particularly set up to stick stuff onto ASDF objects. I suppose if they write to streams that we create, then we can save the stream contents to a string and save them in the operations, though. It will be a burden on writing the test-op menthods, too. 2. Returning a single operation isn't enough, is it? For example, if I have system X, with sub-systems A, B, and C, I may be testing A, B, and C, so my traversal would have to gather up the three subsidiary test-op entities and either package them into the parent test-op object, or rip the test results out of them and push them into the parent test-op object. I don't believe the plan-then-execute logic of ASDF makes this easy, but I may be wrong. I will have to have a look into the latest code and see how the operations are returned. Thanks for the pointer. best, r

Gary King wrote:

Hi Robert,

...

I don't believe that this is a general solution, for two reasons:

I agree that it isn't a general solution especially since there is no interface/API for clients to do anything with an ASDF operation! It might, however, be a small step in the right direction.

An alternative solution would be to provide a :stream or :filename init argument for the test-op operation class and bind a dynamic variable around every perform, making the stream or filename available for writing.... Best, r

Daniel Herring wrote:

On Mon, 5 Oct 2009, Robert Goldman wrote:

...

Gary King wrote:

...

Hi Robert,

...

I don't believe that this is a general solution, for two reasons: I agree that it isn't a general solution especially since there is no interface/API for clients to do anything with an ASDF operation! It might, however, be a small step in the right direction. An alternative solution would be to provide a :stream or :filename init argument for the test-op operation class and bind a dynamic variable around every perform, making the stream or filename available for writing....

Why serialize the data? Could we design a structured API to be used by other tools?

What if each test logged messages to asdf:*test-stream* and finished by calling (asdf:test-result test-description-string result-keyword)

DejaGnu has a good description of test results; the keywords from good to bad could be :xpass, :xfail, :untested, :unresolved, :unsupported, :pass, and :fail. http://www.gnu.org/software/dejagnu/manual/x47.html

Then asdf:test-op could return (values worst-result results-list).

Of course ASDF doesn't need to reinvent testing; there are plenty of existing frameworks to choose from. http://www.cliki.net/test_framework

What I am after is an ASDF test-op that will adapt as well as possible to the widest possible set of testing frameworks. I agree that we should not attempt to reinvent testing. That is why I have been suggesting that we provide a test operation that binds a stream --- because most of the test frameworks I have worked with provide a test report, rather than returning results. I don't believe that having the test-op return a result will work, given the current ASDF execution model. If we are testing system X that has A and B dependencies, then what will happen will be something like .... testing A's components... test-op A....testing B's components ... test-op B .... test other X components ... test-op X It's not obvious to me how we take the results from test-op A and test-op B and roll them up into test-op X. Nor do I see how, in general, a test-op done in LIFT for A, FiveAM for B and NST for X, can be made to communicate with each other. It's not ideal, but I think a separate stream may be about as good as we can do. Best, r

Daniel Herring wrote:

On Mon, 5 Oct 2009, Robert Goldman wrote:

...

Daniel Herring wrote:

...

On Mon, 5 Oct 2009, Robert Goldman wrote:

...

Gary King wrote:

...

Hi Robert,

...

I don't believe that this is a general solution, for two reasons: I agree that it isn't a general solution especially since there is no interface/API for clients to do anything with an ASDF operation! It might, however, be a small step in the right direction. An alternative solution would be to provide a :stream or :filename init argument for the test-op operation class and bind a dynamic variable around every perform, making the stream or filename available for writing.... Why serialize the data? Could we design a structured API to be used by other tools? ... Of course ASDF doesn't need to reinvent testing; there are plenty of existing frameworks to choose from. http://www.cliki.net/test_framework

What I am after is an ASDF test-op that will adapt as well as possible to the widest possible set of testing frameworks. I agree that we should not attempt to reinvent testing.

That is why I have been suggesting that we provide a test operation that binds a stream --- because most of the test frameworks I have worked with provide a test report, rather than returning results.

I don't believe that having the test-op return a result will work, given the current ASDF execution model. ...

Given that, how about deprecating/removing test-op? Then people can continue to simply load sysname-test.asd and run tests however they want.

I see value in creating a widely accepted testing framework. I don't see the benefit in simply binding a stream variable.

Let me clarify. I /do/ see the value of providing a test-op. We have our own testing framework at my company, NST (see presentation hosted at tc-lispers.org for more information). We have provided a back-end for it that hooks into ASDF, so that people can do (test-system <foo>) and our NST tests will run and a report will be printed. This isn't fully ideal, though, since the results of tests will be mingled in the console output together with compilation results, which makes finding the test outputs a little more difficult than it should be. So I think providing a standardized output stream would be helpful, and probably generally helpful, since no matter what your unit-testing library, if it writes a report to the console, and is invoked from ASDF, it's likely to suffer from the same problem of output interleaving. I don't think that ASDF needs to provide a test framework; all it needs to do is to provide hooks for invoking existing test framework(s). Best, r

Attila Lendvai wrote:

...

That is why I have been suggesting that we provide a test operation that binds a stream --- because most of the test frameworks I have worked with provide a test report, rather than returning results.

fyi, stefil returns a CLOS object containing the test results (and provides slime inspector customizations to present it specially in the inspector).

as it doesn't work too well through asdf:test-op, it also provides a print-object method that prints a minimal text representation of the data.

when test-op'ed, we print the result object to *standard-output*. when used interactively, the test defun that was used to start the testing simply returns the result value which we inspect in slime when needed.

Do you see the same problems I see? I find that my test's output can get interleaved with irrelevant chaff. So some of our code has something like this: (defmethod perform :around ((op our-test-op) c) (let ((<special variable> (test-stream op))) (call-next-method))) It's actually far messier than this, because the <special variable>'s package needs to be wrangled, and I needed to do fussy things with make-sub-operation (is there some reason this is not a generic function?) to make sure that our-test-op gets propagated through traverse. Actually, I tell a lie. Getting make-sub-operation to do the right thing was so hard, that I ended up doing something horrible with CHANGE-CLASS. The resulting code was not suitable for sensitive eyes.

Robert Goldman writes:

An alternative solution would be to provide a :stream or :filename init argument for the test-op operation class and bind a dynamic variable around every perform, making the stream or filename available for writing....

To me, the most interesting advantage that I see in ASDF providing a test operation, is that it should allow for automatic testing of arbitrary software packages. I do not see how providing a stream argument is relevant to that. Or do you propose that people should /parse/ a test framework's output? -T.

The problem is that the asdf framework does not lend itself to returning things from ops (particularly if you need to roll up subsidiary results from sub-operations). ___ Robert P. Goldman Principal Scientist, SIFT, LLC www.sift.info ..... Original Message ....... On Thu, 8 Oct 2009 12:18:50 +0200 "Juan Jose Garcia-Ripoll" <juanjose.garciaripoll@googlemail.com> wrote:

On Thu, Oct 8, 2009 at 12:16 PM, Tobias C. Rittweiler <tcr@freebits.de> wrote:

...

To me, the most interesting advantage that I see in ASDF providing a test operation, is that it should allow for automatic testing of arbitrary software packages.

Indeed! See http://ecls.sourceforge.net/logs_lib.html

...

I do not see how providing a stream argument is relevant to that. Or do you propose that people should /parse/ a test framework's output?

Please, please do not force that upon us. It would be so EASY to ask each tester to just return a sequence or number with the tests that failed!

Juanjo

-- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://juanjose.garciaripoll.googlepages.com

_______________________________________________ asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

No, nobody's supposed to parse it. It's just so if you want you can look at *only* the output of the tester, not the output of the tester mooshed together with the output of the compiler. Our tester does a bunch of compilation and esp on sbcl, the test output proper gets mixed together with lots of chaff, so it's hard to find, e.g., which tests are failing... ___ Robert P. Goldman Principal Scientist, SIFT, LLC www.sift.info ...... Original Message ....... On Thu, 08 Oct 2009 12:16:48 +0200 "Tobias C. Rittweiler" <tcr@freebits.de> wrote:

Robert Goldman writes:

...

An alternative solution would be to provide a :stream or :filename init argument for the test-op operation class and bind a dynamic variable around every perform, making the stream or filename available for writing....

To me, the most interesting advantage that I see in ASDF providing a test operation, is that it should allow for automatic testing of arbitrary software packages.

I do not see how providing a stream argument is relevant to that. Or do you propose that people should /parse/ a test framework's output?

-T.

_______________________________________________ asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

Tobias C. Rittweiler wrote:

Robert Goldman writes:

...

2. Returning a single operation isn't enough, is it? For example, if I have system X, with sub-systems A, B, and C, I may be testing A, B, and C, so my traversal would have to gather up the three subsidiary test-op entities and either package them into the parent test-op object, or rip the test results out of them and push them into the parent test-op object. I don't believe the plan-then-execute logic of ASDF makes this easy, but I may be wrong.

Does performing TEST-OP on a sytem really result in testing all the system's dependencies? Or did you mean something else?

We often have large structured systems where testing system X is done by testing subsidiary systems A, B, and C that X depends on. Consider, for example, if one were to write a test-op for CLSQL. One might then have subsidiary systems for the various DB backends, and one would have the test-op for CLSQL run the test-op on each backend (or some subset of the backends that are turned on). best, r

Tobias C. Rittweiler wrote:

Robert Goldman writes:

...

Tobias C. Rittweiler wrote:

...

Robert Goldman writes:

...

2. Returning a single operation isn't enough, is it? For example, if I have system X, with sub-systems A, B, and C, I may be testing A, B, and C, so my traversal would have to gather up the three subsidiary test-op entities and either package them into the parent test-op object, or rip the test results out of them and push them into the parent test-op object. I don't believe the plan-then-execute logic of ASDF makes this easy, but I may be wrong. Does performing TEST-OP on a sytem really result in testing all the system's dependencies? Or did you mean something else? We often have large structured systems where testing system X is done by testing subsidiary systems A, B, and C that X depends on.

Consider, for example, if one were to write a test-op for CLSQL. One might then have subsidiary systems for the various DB backends, and one would have the test-op for CLSQL run the test-op on each backend (or some subset of the backends that are turned on).

So CLSQL's method on TEST-OP should perform tests on each backend and then merge the results of the subsystems into one result, shouldn't it?

That's one way of doing it, I suppose. We typically do something like: (:in-order-to ((test-op (test-op comp1)) (test-op (test-op comp2)) ...)) Yes, one can work around that, and hand-code more, but I'd prefer not to. OTOH, I'm coming to the conclusion that ASDF simply doesn't offer a good enough solution to this problem, and fixing it in ASDF won't benefit enough ASDF users. The different use patterns, and different underlying test libraries seem to diverge enough that we should all fix this ourselves. best, r

Faré wrote:

Maybe ASDF is the wrong place to try to standardize testing infrastructure?

This is the conclusion I have reached, as well. I was hoping that some very weak standard could be arrived at that would make the test-op more generally useful to people installing systems, so that they could simply run the test-op and easily tell whether or not the test operation was successful. However, it may be that it's just a combination of features of our test framework and the way we have built our systems that makes it difficult for us (the original problem --- we find that we get compilation and loading output mixed together with the test output), so this is not a place where the ASDF community can readily get consensus. So please consider the suggest dropped. I may try to come back to the list with a proposal for modifying the MAKE-SUB-OPERATION function so that it makes specializing operations (including the test-op) easier. That would be a more general proposal that might make many things easier for ASDF system specifiers.

I mean, maybe instead the authors of various test infrastructures should have a common list where they discuss interoperability, reporting, and a declarative way of specifying dependencies between test suites, between files and test suites, etc.?

Since for instance XCVB doesn't have testing support yet, but testing support is amongst our eventual goals, I would be eager to have feedback on how a new build system could "do things right" in this regard -- or perhaps avoid doing anything, and just plugging into something else.

Let me suggest a few desiderata: 1. Make it possible for operation performance to return values. Think about what the types of these values should be while designing this facility. 2. Provide a standard means for specifying how sub-operation values should roll up to the values of their parents. 3. If you are taking the plan then act design framework, think about how to keep the (direct) child operations of a parent in some kind of procedure call envelope (dynamic scope extent). 4. Deal with the difference between how different operations should handle values and failure. E.g., a test operation should simply accumulate failures. A build operation should almost always fail at the first point of failure. Hope that helps, r

Tobias C. Rittweiler wrote:

Robert Goldman writes:

...

Faré wrote:

...

Maybe ASDF is the wrong place to try to standardize testing infrastructure? This is the conclusion I have reached, as well. I was hoping that some very weak standard could be arrived at that would make the test-op more generally useful to people installing systems, so that they could simply run the test-op and easily tell whether or not the test operation was successful.

However, it may be that it's just a combination of features of our test framework and the way we have built our systems that makes it difficult for us (the original problem --- we find that we get compilation and loading output mixed together with the test output), so this is not a place where the ASDF community can readily get consensus.

So please consider the suggest dropped.

Please excuse me that I cannot grow at peace with this conclusion.

The missed utility is far too great to let this rest:

Juanjo's automatic regression testing work to check for regressions in ECL is dead on, and I'm sure other implementators would be interested in such infrastructure as well.

The same is true for D Herring and his LibCL project which looks very promising, but is also a candidate for automatic regression testing. I think that Daniel is the right guy for the job and I hope he'll have the necessary perseverance for what he has started.

I have to excuse myself again because I'm not familiar with ASDF in any great detail; on the other hand, ASDF -- unlike most other Lisp project -- consists of over half a dozen of people who do know it pretty well, so I cannot believe that we can not come up with some solution.

In particular because it's my impression that the problems have been over-stated.

I think there are some problems which are real, but perhaps not insuperable: 1. Not all regression test frameworks are functional, returning values. Some write reports instead. 2. One needs to come up with a means of combining operation results that takes into account the structure of the plan that traverse produces and that operate then executes. If in order to test-op X, I must test-op A and B, how do I combine together the test results from A, B, and X into a top-level operation result. Perhaps OPERATION-ANCESTOR can be pressed into service. 3. Is there some way to do this such that the regression-testing framework need not be made aware of ASDF? Ideally, ASDF and the regression-testing framework would be defined independently, and some additional combination code would be defined that would link the two together, and that would be loaded downstream. I'd be reluctant to see a solution that demanded our regression-testing frameworks become dependent on ASDF. I believe that this problem could be easily circumvented by defining new systems like ASDF-FIVEAM, ASDF-NST, ASDF-STEFIL. At my company, we have already been working on ASDF-NST, but it has not reached a fully satisfactory state. best, r

Juan Jose Garcia-Ripoll writes:

On Mon, Oct 19, 2009 at 12:00 AM, Robert Goldman wrote:

[for the record; the topmost paragraph came from me:]

...

...

In particular because it's my impression that the problems have been over-stated.

I think so. And this is indicated by the following paragraphs

...

1.  Not all regression test frameworks are functional, returning values.  Some write reports instead.

That does not mean anything. ASDF may impose one behavior w.r.t to the output and the package maintainer may decide to output T saying "hey, there might be errors, but I do not know." Later on other software and test suites will catch up. With RT, for instance it is trivial to grab the number of failed tests.

...

2.  One needs to come up with a means of combining operation results that takes into account the structure of the plan that traverse produces and that operate then executes.  If in order to test-op X, I must test-op A and B, how do I combine together the test results from A, B, and X into a top-level operation result.  Perhaps OPERATION-ANCESTOR can be pressed into service.

You are just imposing too much complexity. If I want to test package Cl-UNICODE, I do nont want to test FLEXI-STREAMS or U-SOCKETS. Tests should be atomic and not generate a tree of actions like ASDF does not for everything.

As an interim solution, there can be some merge operation which, for the moment, just bails out if multiple results really have to be combined.

...

3.  Is there some way to do this such that the regression-testing framework need not be made aware of ASDF?

No, there is none. Regression tests are libraries which are used by people using ASDF. Eventually they may provide their own operations, extending what ASDF provides, but currently it is better to have some bare minimum from ASDF and let each package maintainer decide how they compute the success of their test suite.

Right. In particular: between ASDF and the regression test framework always stands a _human being_, the maintainer of the software package. It's his duty to interconnect his choice of third-party test framework with ASDF -- that is if he really wants to. If he doesn't, well, he just shouldn't. Existing practise does luckily not matter much in this case. It's about meta-information that nobody can depend on at the moment because it does not exist. And if it exists, it'll matter to people who care and will do the right thing. (Which may include fixing other people's code; but see that it's in _their_ own interests to do so, so they will.)

Blocking this development just because there are 5 test suites and you do not know how to combine them with ASDF is really absurd. ASDF's specifications can not depend on what your company or other people's companies are setting up for their workflow.

Blocking is way too strong a word. After all, both of us want to piggy-back onto work of others. We can just hope to make our case clear to those who contribute code---or contribute ourselves. -T.

Juan Jose Garcia-Ripoll wrote:

On Mon, Oct 19, 2009 at 12:00 AM, Robert Goldman <rpgoldman@sift.info> wrote:

...

...

In particular because it's my impression that the problems have been over-stated.

I think so. And this is indicated by the following paragraphs

[..snip..]

...

2. One needs to come up with a means of combining operation results that takes into account the structure of the plan that traverse produces and that operate then executes. If in order to test-op X, I must test-op A and B, how do I combine together the test results from A, B, and X into a top-level operation result. Perhaps OPERATION-ANCESTOR can be pressed into service.

You are just imposing too much complexity. If I want to test package Cl-UNICODE, I do nont want to test FLEXI-STREAMS or U-SOCKETS. Tests should be atomic and not generate a tree of actions like ASDF does not for everything.

Please see earlier discussion about this topic. The point I made there is that systems may have COMPONENT subsystems such that you want to test the entire system together. Consider, e.g., a DB library with multiple backends, each described in a separate system. In order to do the test-op on the DB library you want to do the test on all active backends. Similarly, McCLIM has multiple graphics display backends, not all of which are loaded at a given time. I work actively on three (or four, depending on how you count) large CL-based applications. Each one of them is made up of multiple ASDF subsystems. The example of CL-UNICODE is a strawman, because it is the case of testing a system and its libraries. There is also the case of testing a system and its subsystems, which is the more interesting one. For that matter, though, if I am a CL-UNICODE /user/, I may very well want to test to see if it will work in its current installation, in which case I /do/ want to test FLEXI-STREAMS and U-SOCKETS, because I want to know whether my installation works. [..snip..]

"Tobias C. Rittweiler" writes:

What if we start with the simple case of only carring one bit of meta-information (successp), merging is nothing more than ORing together the results.

Bah, s/OR/AND/ -T.