Re: [asdf-devel] About regression tests and libraries
Juan Jose Garcia-Ripoll writes:
What I would like to ask CL developers is to begin shipping regression tests with their libraries, and perhaps agree on an standard so that one can programatically check whether a library builds or not. This feature could be even included in ASDF, asdf-install, etc, just like "make check" is now a de-facto standad in Autoconf.
There's (ASDF:OOS 'ASDF:TEST-OP <system>), and on very recent revisions even (ASDF:TEST-SYSTEM <system>) as an abbeviation. I don't think this has been properly documented in the manual of ASDF yet. I hence CC'd the asdf-devel mailing list so this will hopefully get promoted with the due it deserves. -T.
Tobias C. Rittweiler wrote:
Juan Jose Garcia-Ripoll writes:
What I would like to ask CL developers is to begin shipping regression tests with their libraries, and perhaps agree on an standard so that one can programatically check whether a library builds or not. This feature could be even included in ASDF, asdf-install, etc, just like "make check" is now a de-facto standad in Autoconf.
There's (ASDF:OOS 'ASDF:TEST-OP <system>), and on very recent revisions even (ASDF:TEST-SYSTEM <system>) as an abbeviation.
I don't think this has been properly documented in the manual of ASDF yet. I hence CC'd the asdf-devel mailing list so this will hopefully get promoted with the due it deserves.
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for (asdf:test-system <system>) to return a value indicating whether or not <system> has passed its tests. Some time ago, I proposed that ASDF provide a stream argument to the test-op, providing a place into which the testing tool could dump its test report for human inspection. I can't say that this suggestion received universal approbation. Or disapprobation. It merely received near-universal lack of interest! ;-) Related to the question of what the test-op should provide to its invoker is the question of how dependencies should be propagated. Cheers, r
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
FWIW, I recently changed operate-on-system (and therefore asdf:test- system, asdf:load-system, etc.) to return the operation instance that they processed. If ASDF also added and documented (shocking!) a few canonical slots in the test-op class then these could be used to transmit information. As a first pass, I'd suggest test-properties - a property list of test system specific stuff test-result - a keyword from the list :success, :failures, :errors, errors-and-failures
Some time ago, I proposed that ASDF provide a stream argument to the test-op, providing a place into which the testing tool could dump its test report for human inspection. I can't say that this suggestion received universal approbation. Or disapprobation. It merely received near-universal lack of interest! ;-)
Sigh. I think this is a good idea; if we added a test-stream slot to the test-op, then test systems could find it there. A tricky part is that test systems may not want to rely on the existence of ASDF while running. So they'll need some sort of adaptor that adds stuff to the ASDF operation if it exists. Would it make sense to define and export *current-operation* to make this easier?
Related to the question of what the test-op should provide to its invoker is the question of how dependencies should be propagated.
I think that the default is that test-op depends on load-op. other thoughts? -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
On 2009-08-05, at 21:22 , Gary King wrote:
...
Some time ago, I proposed that ASDF provide a stream argument to the test-op, providing a place into which the testing tool could dump its test report for human inspection. I can't say that this suggestion received universal approbation. Or disapprobation. It merely received near-universal lack of interest! ;-)
Sigh. I think this is a good idea; if we added a test-stream slot to the test-op, then test systems could find it there.
A tricky part is that test systems may not want to rely on the existence of ASDF while running. So they'll need some sort of adaptor that adds stuff to the ASDF operation if it exists. Would it make sense to define and export *current-operation* to make this easier?
Related to the question of what the test-op should provide to its invoker is the question of how dependencies should be propagated.
I think that the default is that test-op depends on load-op.
other thoughts?
mix stream delegation into the operation based on the slot value.
Hi James,
mix stream delegation into the operation based on the slot value.
I don't think I understand. Can you provide a pseudo-example? thanks, -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
Gary King wrote:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
FWIW, I recently changed operate-on-system (and therefore asdf:test-system, asdf:load-system, etc.) to return the operation instance that they processed.
If ASDF also added and documented (shocking!) a few canonical slots in the test-op class then these could be used to transmit information. As a first pass, I'd suggest
test-properties - a property list of test system specific stuff test-result - a keyword from the list :success, :failures, :errors, errors-and-failures
Some time ago, I proposed that ASDF provide a stream argument to the test-op, providing a place into which the testing tool could dump its test report for human inspection. I can't say that this suggestion received universal approbation. Or disapprobation. It merely received near-universal lack of interest! ;-)
Sigh. I think this is a good idea; if we added a test-stream slot to the test-op, then test systems could find it there.
A tricky part is that test systems may not want to rely on the existence of ASDF while running. So they'll need some sort of adaptor that adds stuff to the ASDF operation if it exists. Would it make sense to define and export *current-operation* to make this easier?
You'd still need the asdf stuff to get at this. The alternative would be to define a special variable that gives direct access to the stream, which is not an asdf-dependent entity. The problem with that approach is that you're pretty much doomed to crap up the CL-USER namespace.
Related to the question of what the test-op should provide to its invoker is the question of how dependencies should be propagated.
I think that the default is that test-op depends on load-op.
Right, but what about testing system <foo> --- do we test <foo>'s depended on systems? My guess is that the correct answer to this is "it depends." If you have a standard set of libraries, the answer will typically be "no." OTOH, if you have a very large application (e.g., a web site with many components providing different features), you may well want to test the subsystems that are part of the top-level system.
other thoughts?
We have found in practice that we end up creating a test-<foo> system definition to complement our <foo> system definitions. The reason for this is that the existing ASDF structures have only components that are unconditional. For the test op we typically want to load new files. A second item is that the test system typically :depends-on the test framework library. This suggests a question: is there a nice way to specify additional files that need to be loaded ONLY for the test-op? I can think of some ways to achieve this with the existing ASDF framework, but they would be cumbersome and non-standard. Would it be possible to simply add a :test-components argument to defsystem? Note that the use of additional <test>-foo systems also helps us tailor subcomponent testing. Another note --- I found in an earlier attempt to tweak the test-op that it worked poorly to subclass operations (as opposed to components). IIRC, this was because of the way dependent operation instances are built. Alas, I didn't take good notes when I did this, and have garbage-collected much of what I learned. r
Gary King wrote:
I think that the default is that test-op depends on load-op.
I'm looking at asdf.lisp right now, and I don't see any such dependency. I agree that this would be A Good Thing. It's really a pain to have to add (:in-order-to ((test-op (load-op <sysname>))) to all the systems with testers... Here's a patch that adds that and also adds some more information to an error I tripped over while writing the first patch.... Best, r P.S. did you have a chance to push my patch to document weakly-depends-on? diff --git a/asdf.lisp b/asdf.lisp index 1e1118d..6d84d02 100644 --- a/asdf.lisp +++ b/asdf.lisp @@ -900,7 +900,7 @@ the head of the tree")) forced (do-one-dep op (second d) (third d)))) (t - (error "Dependencies must be (:version <version>), (:feature <feature>), or a name")))) + (error "Bad dependency ~a. Dependencies must be (:version <version>), (:feature <feature>), or a name" d)))) (t (appendf forced (do-one-dep op d nil))))))))) (aif (component-visited-p operation c) @@ -1122,6 +1122,9 @@ the head of the tree")) (defclass test-op (operation) ()) +(defmethod component-depends-on :around ((o test-op) (c system)) + (cons `(load-op ,(component-name c)) (call-next-method))) + (defmethod perform ((operation test-op) (c component)) nil) @@ -1132,6 +1135,7 @@ the head of the tree")) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; invoking operations + (defun operate (operation-class system &rest args &key (verbose t) version force &allow-other-keys) (declare (ignore force))
Hi Robert (and all), I've been wanting to comment on this (and other) threads but life/work are hammering pretty hard right now.
I'm looking at asdf.lisp right now, and I don't see any such dependency.
right you are, I was confusing this with (of all things)
(defmethod operation-done-p ((operation test-op) (c system)) "Testing a system is _never_ done." nil)
Here's a patch that adds that and also adds some more information to an error I tripped over while writing the first patch....
Great!
P.S. did you have a chance to push my patch to document weakly- depends-on?
Yes, ages (so to speak) ago. I'll apply your patch shortly, -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted... Cheers, -- Nikodemus
Nikodemus Siivola wrote:
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
I suppose, but typically a regression test suite should attempt to run ALL of its tests, collecting a set of failing tests, and then returning that information. And demanding interactive attention ("I'm going to throw you into the debugger if I fail") may also be undesirable in practice. I suppose handler-case would be enough for that, but I'm still uncomfortable with the idea. Best, r
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
I suppose, but typically a regression test suite should attempt to run ALL of its tests, collecting a set of failing tests, and then returning that information. And demanding interactive attention ("I'm going to throw you into the debugger if I fail") may also be undesirable in practice. I suppose handler-case would be enough for that, but I'm still uncomfortable with the idea.
My rationale for suggesting this was that there are two mental modes for running tests: as a user ("Do I have version that seems to be working?") and as a developer ("What did I break now?"). I would expect TEST-SYSTEM to be used mostly by the end users of ASDF systems, whereas presumably developers run their own test-suites more directly -- I'm assuming that the typical TEST-OP definition would look like this: (defmethod perform ((op test-op) (sys (eql (find-system :my-system)))) (foo-unit:run-tests :my-system-tests :on-failure :error)) and developers of MY-SYSTEM would normally use RUN-TESTS directly, taking advantage of whatever reporting and debugging features the test-suite provides. Cheers, -- Nikodemus
Nikodemus Siivola wrote:
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
I suppose, but typically a regression test suite should attempt to run ALL of its tests, collecting a set of failing tests, and then returning that information. And demanding interactive attention ("I'm going to throw you into the debugger if I fail") may also be undesirable in practice. I suppose handler-case would be enough for that, but I'm still uncomfortable with the idea.
My rationale for suggesting this was that there are two mental modes for running tests: as a user ("Do I have version that seems to be working?") and as a developer ("What did I break now?").
I would expect TEST-SYSTEM to be used mostly by the end users of ASDF systems, whereas presumably developers run their own test-suites more directly -- I'm assuming that the typical TEST-OP definition would look like this:
(defmethod perform ((op test-op) (sys (eql (find-system :my-system)))) (foo-unit:run-tests :my-system-tests :on-failure :error))
and developers of MY-SYSTEM would normally use RUN-TESTS directly, taking advantage of whatever reporting and debugging features the test-suite provides.
I see. However, I think it would be a mistake to force /anyone/ to encounter an immediate error on running the test-op as in the snippet above. I work on a couple of large systems where, in fact, the developers run their own test suites through ASDF, because many of them are not familiar with the test tool (maybe they are working on the java side). They don't want to encounter the lisp debugger. For that matter, we actually package up the results of the tests into an XML file that can be slurped by junit, and assembled into an overall test result. In fact, in some cases there isn't even someone available to interact with the build process --- it's run at a high level by scripts. For relatively small libraries, doing one thing, the "error immediately" behavior might be appropriate. But for large, complex artifacts (e.g., a web site), this behavior is not appropriate. It would be appropriate to /offer/ error-immediately behavior, but not to offer /only/ that behavior. Best, Robert Best, R
Nikodemus Siivola wrote:
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
Cheers,
-- Nikodemus
A second point is that test libraries are often very chatty, and it's unpleasant to have their output mixed in the same stream with the output of the compiler and whatever all else ASDF might happen to trigger splatting onto the output stream. So I really think that some stream argument is essential, even if we were to accept your proposal. best, r
Sorry to parasite this mailing-list, but I'm trying to design the testing aspect of XCVB right now. The constraint is that we want to be able to use make as a backend, and that test results should thus be reified as files if we want to avoid re-running already-computed tests. My dim ideas about it include the following: * test reports are targets depending on properly compiled fasls or images, produce test reports. * a test report is a file the first form of which is a simple SEXP to be read, following some standardized structure to specify overall success, status of individual tests (including error message, maybe information as to last success, etc.). * from the test reports, a success witness (empty file) may be created on success. * the success witness fails to be created when the test wasn't a success, triggering an error for make to catch. * preparing the makefile erases the success witnesses. If ASDF's test-op is to be standardized, may I suggest that we agree on the format for a SEXP to be returned, that will report the test results? [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] A child of five would understand this. Send someone to fetch a child of five. -- Groucho Marx 2009/8/5 Robert Goldman <rpgoldman@sift.info>:
Nikodemus Siivola wrote:
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
Cheers,
-- Nikodemus
A second point is that test libraries are often very chatty, and it's unpleasant to have their output mixed in the same stream with the output of the compiler and whatever all else ASDF might happen to trigger splatting onto the output stream. So I really think that some stream argument is essential, even if we were to accept your proposal.
best, r
Faré writes:
Sorry to parasite this mailing-list, but I'm trying to design the testing aspect of XCVB right now. The constraint is that we want to be able to use make as a backend, and that test results should thus be reified as files if we want to avoid re-running already-computed tests.
Are you sure that XCVB is the right level for that? Shouldn't that be the duty of your test framework? Is XCVB implemented on top of some sort of general dependency graph management library? -T.
Faré wrote:
Sorry to parasite this mailing-list, but I'm trying to design the testing aspect of XCVB right now. The constraint is that we want to be able to use make as a backend, and that test results should thus be reified as files if we want to avoid re-running already-computed tests.
My dim ideas about it include the following: * test reports are targets depending on properly compiled fasls or images, produce test reports. * a test report is a file the first form of which is a simple SEXP to be read, following some standardized structure to specify overall success, status of individual tests (including error message, maybe information as to last success, etc.). * from the test reports, a success witness (empty file) may be created on success. * the success witness fails to be created when the test wasn't a success, triggering an error for make to catch. * preparing the makefile erases the success witnesses.
If ASDF's test-op is to be standardized, may I suggest that we agree on the format for a SEXP to be returned, that will report the test results?
Based on our experience in a large, multi-language framework, I'd suggest a related enhancement: * specify a translation from the s-expression test result format to XML that complies with what junit expects. In collaboration with folks from BBN, my colleague John Maraist has built this capability into his NST test framework. This means that a large system with java components can all be tested together. best, r
Nikodemus Siivola writes:
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
You wrote "tests do not pass"; just to make sure, the plural was intentional, right? You did /not/ mean it should signal an error for each failing test, did you? If I'm running the TEST-OP on some system (with the user, not developer, hat on my head), I'd like to get a nice summary about all failing tests to give a sense about the state of the software. I'm not interested in the minutae of the tests themselves. Signalling an ASDF:TEST-OP-FAILED condition, perhaps even making that hidden in some function, so people can just write (ASDF:TEST-OP-FAIL) within their PERFORM method, is sure an idea if that better fits the architecture of ASDF. -T.
Tobias C. Rittweiler wrote:
Nikodemus Siivola writes:
2009/8/5 Robert Goldman <rpgoldman@sift.info>:
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests. I would propose that as a matter of good style implementations of TEST-OP should signal an error is tests do not pass. It is the simplest way to make sure the result is not misinterpreted...
You wrote "tests do not pass"; just to make sure, the plural was intentional, right? You did /not/ mean it should signal an error for each failing test, did you?
If I'm running the TEST-OP on some system (with the user, not developer, hat on my head), I'd like to get a nice summary about all failing tests to give a sense about the state of the software. I'm not interested in the minutae of the tests themselves.
Signalling an ASDF:TEST-OP-FAILED condition, perhaps even making that hidden in some function, so people can just write (ASDF:TEST-OP-FAIL) within their PERFORM method, is sure an idea if that better fits the architecture of ASDF.
This would actually accommodate itself nicely to the non-value-returning structure of ASDF. A remaining trouble: because of the flat plan-then-execute nature of ASDF, I don't believe the stack contents while performing a test-op lends itself to accumulating together results of component tests. Consider a case where we have a system A and in order to do a TEST-OP on that system, one needs to do a TEST-OP on B and C. I don't believe that (PERFORM TEST-OP A) will be on the stack when doing (PERFORM 'TEST-OP B), so that there's no obvious way to grab up the test results from B and accumulate them inside the TEST-OP-FAILED condition object for A. If I understand correctly, TRAVERSE does a postorder traversal of the tree of systems below A, meaning that (PERFORM TEST-OP A) will be called after (PERFORM TEST-OP B) and (PERFORM TEST-OP C) have exited. Best, r
Robert Goldman writes:
Tobias C. Rittweiler wrote:
There's (ASDF:OOS 'ASDF:TEST-OP <system>), and on very recent revisions even (ASDF:TEST-SYSTEM <system>) as an abbeviation.
I don't think this has been properly documented in the manual of ASDF yet. I hence CC'd the asdf-devel mailing list so this will hopefully get promoted with the due it deserves.
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
What is the reason that OPERATE can not return the return value of PERFORM? -T.
On Sat, Aug 8, 2009 at 6:27 PM, Tobias C. Rittweiler<tcr@freebits.de> wrote:
Robert Goldman writes:
Tobias C. Rittweiler wrote:
There's (ASDF:OOS 'ASDF:TEST-OP <system>), and on very recent revisions even (ASDF:TEST-SYSTEM <system>) as an abbeviation.
I don't think this has been properly documented in the manual of ASDF yet. I hence CC'd the asdf-devel mailing list so this will hopefully get promoted with the due it deserves.
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
What is the reason that OPERATE can not return the return value of PERFORM?
None, but the problem is that each library and developer has his/her own idea of what that output value should be. Alexandria uses NIL for success, CL-PPCRE the opposite. It is all a mess. Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://juanjose.garciaripoll.googlepages.com
Tobias C. Rittweiler wrote:
Robert Goldman writes:
Tobias C. Rittweiler wrote:
There's (ASDF:OOS 'ASDF:TEST-OP <system>), and on very recent revisions even (ASDF:TEST-SYSTEM <system>) as an abbeviation.
I don't think this has been properly documented in the manual of ASDF yet. I hence CC'd the asdf-devel mailing list so this will hopefully get promoted with the due it deserves.
Part of the problem with test-op is that the desired behavior has not been specified by the ASDF community. Because of the nature of ASDF, it is impossible for
(asdf:test-system <system>)
to return a value indicating whether or not <system> has passed its tests.
What is the reason that OPERATE can not return the return value of PERFORM?
At any rate, right now OPERATE does not return the return value of PERFORM. It returns the operation. [I believe someone suggested stashing test success/failure information on the operation in the course of this discussion.] Interestingly, OPERATE seems to go out of its way to discard the return value of PERFORMs (see ll 1152-1153 of ASDF in git). Given the plan-then-perform structure of ASDF, I'm not sure it's entirely obvious how to restructure it so that values are appropriately returned. It's not clear to me how values of subsidiary tests (e.g., other test operations invoked by a top-level TEST-OP) would be combined into values for the top level operation, since there's no result-catching stack frame corresponding to the top-level operation (this is for the same reason that it's tricky to, e.g., use ASDF to wrap operations on a system in a dynamic variable binding). I'm not saying that this couldn't be done, but it's not a simple change to make ASDF do this.
participants (7)
-
Faré -
Gary King -
james anderson -
Juan Jose Garcia-Ripoll -
Nikodemus Siivola -
Robert Goldman -
Tobias C. Rittweiler