We use ASDF to encapsulate the building and testing many systems under an automated test runner.
For a given system, ASDF:TEST-SYSTEM always returns boolean truth as long as the invocation of the underlying test suite succeeds.
This means there is no programatic way to provide a boolean as to whether all the tests passed or not on the invocation of a test suite to the invoker of ASDF:TEST-SYSTEM.
It seems that this issue has been raised before, as the ASDF manual documents TEST-OP https://www.common-lisp.net/project/asdf/asdf/Predefined-operations-of-ASDF.html#Predefined-operations-of-ASDF with:
The results of this operation are not defined by ASDF. It has proven difficult to define how the test operation should signal its results to the user in a way that is compatible with all of the various test libraries and test techniques in use in the community, and given the fact that ASDF operations do not return a value indicating success or failure. For those willing to go to the effort, we suggest defining conditions to signal when a test-op fails, and storing in those conditions information that describes which tests fail.
Is this still the best current practice to introspect the situation of failing tests? Can someone point me to an example implementation of this technique?
I have pushed the system `fiveam-asdf`, which supports integration between ASDF and the FIVEAM test library, to the contribs directory in the ASDF repo. Please have a look.
Be warned! It is old, and not being broke, hasn't been fixed. It inappropriately is housed in the `ASDF` package and inappropriately exports extensions from that package.
But I believe it's still useful as an example of how to raise conditions when the test operation goes wrong. Catching those exceptions can be used to cause a build to fail in a CI system, typically by running lisp in batch mode and having it exit with a nonzero error code if the test operation fails.
Best, r
On 5 Jun 2018, at 4:09, Mark Evenson wrote:
We use ASDF to encapsulate the building and testing many systems under an automated test runner.
For a given system, ASDF:TEST-SYSTEM always returns boolean truth as long as the invocation of the underlying test suite succeeds.
This means there is no programatic way to provide a boolean as to whether all the tests passed or not on the invocation of a test suite to the invoker of ASDF:TEST-SYSTEM.
It seems that this issue has been raised before, as the ASDF manual documents TEST-OP https://www.common-lisp.net/project/asdf/asdf/Predefined-operations-of-ASDF.html#Predefined-operations-of-ASDF with:
The results of this operation are not defined by ASDF. It has proven difficult to define how the test operation should signal its results to the user in a way that is compatible with all of the various test libraries and test techniques in use in the community, and given the fact that ASDF operations do not return a value indicating success or failure. For those willing to go to the effort, we suggest defining conditions to signal when a test-op fails, and storing in those conditions information that describes which tests fail.
Is this still the best current practice to introspect the situation of failing tests? Can someone point me to an example implementation of this technique?
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Jun 5, 2018, at 18:44, Robert Goldman rpgoldman@sift.info wrote:
I have pushed the system fiveam-asdf, which supports integration between ASDF and the FIVEAM test library, to the contribs directory in the ASDF repo. Please have a look.
Be warned! It is old, and not being broke, hasn't been fixed. It inappropriately is housed in the ASDF package and inappropriately exports extensions from that package.
But I believe it's still useful as an example of how to raise conditions when the test operation goes wrong. Catching those exceptions can be used to cause a build to fail in a CI system, typically by running lisp in batch mode and having it exit with a nonzero error code if the test operation fails.
Robert,
Thanks ever so much for [releasing the fiveam-asdf code][1].
[1]: https://gitlab.common-lisp.net/asdf/asdf/tree/master/contrib/fiveam-asdf
I seemingly misunderstand how to use the :CLASS argument to ASDF:DEFSYSTEM, as I cannot quite get your code to work.
If you get a chance, can you eyeball [my example code to try to use FIVEAM-ASDF][2] to tell me if that looks like the correct usage?
[2]: https://github.com/emotiq/asdf-test-harness/blob/master/eg/fiveam-asdf-examp...
Various versions of ASDF-3.1.x (SBCL, CCL, ABCL) are failing with problems about symbols:
Error while trying to load definition for system fiveam-asdf-example from pathname /Users/evenson/work/asdf-test-harness/eg/fiveam-asdf-example.asd:
EXPORT ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM causes name-conflicts in #<PACKAGE "ASDF/USER"> between the following symbols: ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM, ASDF/USER::FIVEAM-TESTER-SYSTEM [Condition of type ASDF/FIND-SYSTEM:LOAD-SYSTEM-DEFINITION-ERROR]
On 6 Jun 2018, at 11:37, Mark Evenson wrote:
On Jun 5, 2018, at 18:44, Robert Goldman rpgoldman@sift.info wrote:
I have pushed the system fiveam-asdf, which supports integration between ASDF and the FIVEAM test library, to the contribs directory in the ASDF repo. Please have a look.
Be warned! It is old, and not being broke, hasn't been fixed. It inappropriately is housed in the ASDF package and inappropriately exports extensions from that package.
But I believe it's still useful as an example of how to raise conditions when the test operation goes wrong. Catching those exceptions can be used to cause a build to fail in a CI system, typically by running lisp in batch mode and having it exit with a nonzero error code if the test operation fails.
Robert,
Thanks ever so much for [releasing the fiveam-asdf code][1].
I seemingly misunderstand how to use the :CLASS argument to ASDF:DEFSYSTEM, as I cannot quite get your code to work.
If you get a chance, can you eyeball [my example code to try to use FIVEAM-ASDF][2] to tell me if that looks like the correct usage?
Various versions of ASDF-3.1.x (SBCL, CCL, ABCL) are failing with problems about symbols:
Error while trying to load definition for system fiveam-asdf-example from pathname /Users/evenson/work/asdf-test-harness/eg/fiveam-asdf-example.asd:
EXPORT ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM causes name-conflicts in #<PACKAGE "ASDF/USER"> between the following symbols: ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM, ASDF/USER::FIVEAM-TESTER-SYSTEM [Condition of type ASDF/FIND-SYSTEM:LOAD-SYSTEM-DEFINITION-ERROR]
I had problems like this, too -- it's because of the fact that when the `defsystem` form is first read, *before* the `:defsystem-depends-on` is loaded, the symbol named "FIVEAM-TESTER-SYSTEM" gets interned in `ASDF/USER` and then when it's later exported from `ASDF/INTERFACE` (used by `ASDF/USER`) it collides with the earlier-read symbol.
I believe that the correct fix for this is to use any new symbols (like `fiveam-tester-system`) in the keyword package, so ``` :class :fiveam-tester-system ``` and then when the `defsystem` form is processed *after* the defsystem dependencies (in this case, `fiveam-asdf`), ASDF will look for keyword symbols in the current package.
Give that a try and see if it fixes your problem.
We should check and see if the processing of keyword symbols is documented clearly in the manual. Possibly there should be a FAQ entry for this issue....
Best, Robert
One more thing. I *believe* that all `fiveam-test-system`s automagically depend on `fiveam`. If I'm wrong about that, I should fix it....
Cheers, r
On 6 Jun 2018, at 12:14, Robert Goldman wrote:
On 6 Jun 2018, at 11:37, Mark Evenson wrote:
On Jun 5, 2018, at 18:44, Robert Goldman rpgoldman@sift.info wrote:
I have pushed the system fiveam-asdf, which supports integration between ASDF and the FIVEAM test library, to the contribs directory in the ASDF repo. Please have a look.
Be warned! It is old, and not being broke, hasn't been fixed. It inappropriately is housed in the ASDF package and inappropriately exports extensions from that package.
But I believe it's still useful as an example of how to raise conditions when the test operation goes wrong. Catching those exceptions can be used to cause a build to fail in a CI system, typically by running lisp in batch mode and having it exit with a nonzero error code if the test operation fails.
Robert,
Thanks ever so much for [releasing the fiveam-asdf code][1].
I seemingly misunderstand how to use the :CLASS argument to ASDF:DEFSYSTEM, as I cannot quite get your code to work.
If you get a chance, can you eyeball [my example code to try to use FIVEAM-ASDF][2] to tell me if that looks like the correct usage?
Various versions of ASDF-3.1.x (SBCL, CCL, ABCL) are failing with problems about symbols:
Error while trying to load definition for system fiveam-asdf-example from pathname /Users/evenson/work/asdf-test-harness/eg/fiveam-asdf-example.asd:
EXPORT ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM causes name-conflicts in #<PACKAGE "ASDF/USER"> between the following symbols: ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM, ASDF/USER::FIVEAM-TESTER-SYSTEM [Condition of type ASDF/FIND-SYSTEM:LOAD-SYSTEM-DEFINITION-ERROR]
I had problems like this, too -- it's because of the fact that when the `defsystem` form is first read, *before* the `:defsystem-depends-on` is loaded, the symbol named "FIVEAM-TESTER-SYSTEM" gets interned in `ASDF/USER` and then when it's later exported from `ASDF/INTERFACE` (used by `ASDF/USER`) it collides with the earlier-read symbol.
I believe that the correct fix for this is to use any new symbols (like `fiveam-tester-system`) in the keyword package, so
:class :fiveam-tester-systemand then when the `defsystem` form is processed *after* the defsystem dependencies (in this case, `fiveam-asdf`), ASDF will look for keyword symbols in the current package.
Give that a try and see if it fixes your problem.
We should check and see if the processing of keyword symbols is documented clearly in the manual. Possibly there should be a FAQ entry for this issue....
Best, Robert
On Jun 6, 2018, at 19:14, Robert Goldman rpgoldman@sift.info wrote:
[…]
If you get a chance, can you eyeball [my example code to try to use FIVEAM-ASDF][2] to tell me if that looks like the correct usage?
Various versions of ASDF-3.1.x (SBCL, CCL, ABCL) are failing with problems about symbols:
Error while trying to load definition for system fiveam-asdf-example from pathname /Users/evenson/work/asdf-test-harness/eg/fiveam-asdf-example.asd:
EXPORT ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM causes name-conflicts in #<PACKAGE "ASDF/USER"> between the following symbols: ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM, ASDF/USER::FIVEAM-TESTER-SYSTEM [Condition of type ASDF/FIND-SYSTEM:LOAD-SYSTEM-DEFINITION-ERROR]
I had problems like this, too -- it's because of the fact that when the defsystem form is first read, before the :defsystem-depends-on is loaded, the symbol named "FIVEAM-TESTER-SYSTEM" gets interned in ASDF/USER and then when it's later exported from ASDF/INTERFACE (used by ASDF/USER) it collides with the earlier-read symbol.
I believe that the correct fix for this is to use any new symbols (like fiveam-tester-system) in the keyword package, so
:class :fiveam-tester-system
and then when the defsystem form is processed after the defsystem dependencies (in this case, fiveam-asdf), ASDF will look for keyword symbols in the current package.
Give that a try and see if it fixes your problem.
Unfortunately, it doesn’t help to specify a keyword, but one gets a new error:
Error while trying to load definition for system asdf-test-harness-example from pathname /Users/evenson/work/asdf-test-harness/eg/asdf-test-harness-example.asd:
The following symbols need to be imported to #<Package "ASDF/INTERFACE"> before they can be exported from that package: (:HARNESS-TEST):
I think with some futzing around with EVAL-WHEN around EXPORT statements we can get this to work.
I’m coming to the end of my day with a Common Lisp Foundation board meeting, so I will probably get back to this tomorrow.
best, Mark
Will you please send me the example (with ancillary files), so that I can see exactly what's going wrong?
Most of my actual systems reverted to using ``` (asdf:load-system "fiveam-asdf") ``` in the `.asd` file, because I didn't know about the keyword trick, so I haven't tested it extensively.
Also, I tend *not* to use the `ASDF/USER` package, but instead be old school and make a package for each `.asd` file.
Best, r
On 6 Jun 2018, at 13:45, Mark Evenson wrote:
On Jun 6, 2018, at 19:14, Robert Goldman rpgoldman@sift.info wrote:
[…]
If you get a chance, can you eyeball [my example code to try to use FIVEAM-ASDF][2] to tell me if that looks like the correct usage?
Various versions of ASDF-3.1.x (SBCL, CCL, ABCL) are failing with problems about symbols:
Error while trying to load definition for system fiveam-asdf-example from pathname /Users/evenson/work/asdf-test-harness/eg/fiveam-asdf-example.asd:
EXPORT ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM causes name-conflicts in #<PACKAGE "ASDF/USER"> between the following symbols: ASDF/INTERFACE::FIVEAM-TESTER-SYSTEM, ASDF/USER::FIVEAM-TESTER-SYSTEM [Condition of type ASDF/FIND-SYSTEM:LOAD-SYSTEM-DEFINITION-ERROR]
I had problems like this, too -- it's because of the fact that when the defsystem form is first read, before the :defsystem-depends-on is loaded, the symbol named "FIVEAM-TESTER-SYSTEM" gets interned in ASDF/USER and then when it's later exported from ASDF/INTERFACE (used by ASDF/USER) it collides with the earlier-read symbol.
I believe that the correct fix for this is to use any new symbols (like fiveam-tester-system) in the keyword package, so
:class :fiveam-tester-system
and then when the defsystem form is processed after the defsystem dependencies (in this case, fiveam-asdf), ASDF will look for keyword symbols in the current package.
Give that a try and see if it fixes your problem.
Unfortunately, it doesn’t help to specify a keyword, but one gets a new error:
Error while trying to load definition for system asdf-test-harness-example from pathname /Users/evenson/work/asdf-test-harness/eg/asdf-test-harness-example.asd:
The following symbols need to be imported to #<Package "ASDF/INTERFACE"> before they can be exported from that package: (:HARNESS-TEST):
I think with some futzing around with EVAL-WHEN around EXPORT statements we can get this to work.
I’m coming to the end of my day with a Common Lisp Foundation board meeting, so I will probably get back to this tomorrow.
best, Mark
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Jun 6, 2018, at 21:08, Robert Goldman rpgoldman@sift.info wrote:
Will you please send me the example (with ancillary files), so that I can see exactly what's going wrong?
[…]
Attaching
wget https://raw.githubusercontent.com/emotiq/asdf-test-harness/master/eg/fiveam-...
On Jun 6, 2018, at 22:41, Mark Evenson evenson@panix.com wrote:
On Jun 6, 2018, at 21:08, Robert Goldman rpgoldman@sift.info wrote:
Will you please send me the example (with ancillary files), so that I can see exactly what's going wrong?
wget https://raw.githubusercontent.com/emotiq/asdf-test-harness/master/eg/lisp-un...
When I replaced `fiveam-tester-system` with `:fiveam-tester-system` or `"FIVEAM-TESTER-SYSTEM"`, the load went fine for me with the latest ASDF on Allegro, SBCL, and ABCL 1.5.0.
Really, no one should be running with an ASDF that's older than 3.2 these days. I thought that all the OSS lisps were distributed with at least 3.2 now. Why are you running with 3.1.x?
Best, r
On Jun 7, 2018, at 00:21, Robert Goldman rpgoldman@sift.info wrote:
When I replaced fiveam-tester-system with :fiveam-tester-system or "FIVEAM-TESTER-SYSTEM", the load went fine for me with the latest ASDF on Allegro, SBCL, and ABCL 1.5.0.
I must have been somehow introducing errors by being non-rigorous in my testing yesterday because when I woke up fresh today to revisit this problem, I was indeed able to get both the keyword package designator and the string package designator forms to work under sbcl-1.4.8, ccl-1.11.5, and lispworks-pro-7.1.0.0.
I worked over my code to the point that I am ready to start integrating with my test harness invocation code.
Instead of using ERROR for raising the condition, I am [going to use SIGNAL][1]. This will mean that as long as no handler for ASDF:TEST-SUITE-FAILURES is installed, the test suite can be invoked by a developer normally. But when the automated test runner invokes the tests with an [appropiate handler installed][2], I’ll be able to make the test invocation “go red."
[1]: https://github.com/emotiq/asdf-test-harness/blob/master/asdf-test-harness.li...
[2]: https://github.com/emotiq/asdf-test-harness/blob/master/eg/run-with-handler....
Really, no one should be running with an ASDF that's older than 3.2 these days. I thought that all the OSS lisps were distributed with at least 3.2 now. Why are you running with 3.1.x?
I’m attempting to support a rather diverse set of Lisp hackers using a fair mix of development environments so it is organizationally easier for me to use the built-in ASDF. asdf-3.1.1 is what ccl-1.11.5 ships with, so it is one of versions I would like to support.
My intention for ASDF-TEST-HARNESS will be to provide a single mechanism to support test invocation for variety of differing test frameworks. The necessary [ASDF definition][3] will be quite simple, merely naming the test framework, and the package to test:
(defsystem "asdf-test-harness-example" :defsystem-depends-on (asdf-test-harness) :class :harness-test :test-framework :lisp-unit :test-package :lisp-unit-example :components ((:file "lisp-unit-example")))
[3]: https://github.com/emotiq/asdf-test-harness/blob/master/eg/asdf-test-harness...
Almost all test frameworks which I have reviewed support a method to introspect a package for running the tests contained therein. All one will have to do to support a new framework will be write a specialization on a generic method which given the package name, runs the tests, collects the results, and returns a boolean truth value as to whether that test invocation should be considered pass (“Green”) or fail (“Red”).
Initially, I plan to support LISP-UNIT, PROVE, and FIVEAM.
Now, if I can only get Quicklisp to work again with :DEFSYSTEM-DEPENDS-ON clauses, I will be a pig in slop heaven…
Thanks very much for the help.
Instead of using ERROR for raising the condition, I am [going to use SIGNAL][1]. This will mean that as long as no handler for ASDF:TEST-SUITE-FAILURES is installed, the test suite can be invoked by a developer normally. But when the automated test runner invokes the tests with an [appropiate handler installed][2], I’ll be able to make the test invocation “go red."
The manual ought to more strongly recommend using ERROR for raising the condition. That's the one and only interface to warn casual users that tests failed. Failing tests that fail to throw ERROR fail the user in an even worse way.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs." — Nikodemus Siivola
On 7 Jun 2018, at 1:03, Mark Evenson wrote:
On Jun 7, 2018, at 00:21, Robert Goldman rpgoldman@sift.info wrote:
When I replaced fiveam-tester-system with :fiveam-tester-system or "FIVEAM-TESTER-SYSTEM", the load went fine for me with the latest ASDF on Allegro, SBCL, and ABCL 1.5.0.
I must have been somehow introducing errors by being non-rigorous in my testing yesterday because when I woke up fresh today to revisit this problem, I was indeed able to get both the keyword package designator and the string package designator forms to work under sbcl-1.4.8, ccl-1.11.5, and lispworks-pro-7.1.0.0.
I worked over my code to the point that I am ready to start integrating with my test harness invocation code.
Instead of using ERROR for raising the condition, I am [going to use SIGNAL][1]. This will mean that as long as no handler for ASDF:TEST-SUITE-FAILURES is installed, the test suite can be invoked by a developer normally. But when the automated test runner invokes the tests with an [appropiate handler installed][2], I’ll be able to make the test invocation “go red."
This sounds like an improvement. I may import it into the main version of fiveam-asdf.
Really, no one should be running with an ASDF that's older than 3.2 these days. I thought that all the OSS lisps were distributed with at least 3.2 now. Why are you running with 3.1.x?
I’m attempting to support a rather diverse set of Lisp hackers using a fair mix of development environments so it is organizationally easier for me to use the built-in ASDF. asdf-3.1.1 is what ccl-1.11.5 ships with, so it is one of versions I would like to support.
My intention for ASDF-TEST-HARNESS will be to provide a single mechanism to support test invocation for variety of differing test frameworks. The necessary [ASDF definition][3] will be quite simple, merely naming the test framework, and the package to test:
(defsystem "asdf-test-harness-example" :defsystem-depends-on (asdf-test-harness) :class :harness-test :test-framework :lisp-unit :test-package :lisp-unit-example :components ((:file "lisp-unit-example")))
That sounds interesting. Note for some bigger systems we have had test operations that work over multiple packages (hence the longer form for the specification in the fiveam-tester-system class.
Almost all test frameworks which I have reviewed support a method to introspect a package for running the tests contained therein. All one will have to do to support a new framework will be write a specialization on a generic method which given the package name, runs the tests, collects the results, and returns a boolean truth value as to whether that test invocation should be considered pass (“Green”) or fail (“Red”).
OK, but note that you need a means of handling cases where running one `test-system` involves testing multiple subsidiary systems. In which case the boolean return will have to be handled carefully, because presumably you want your overall test to have `AND` logical semantics. You'll have to signal component system test failures and then have a handler that will defer them until all the tests are finally run.
Initially, I plan to support LISP-UNIT, PROVE, and FIVEAM.
This sounds great. We could talk about whether to continue to use asdf/contribs/ for this. Maybe not, since you will presumably want to maintain this code in your own repo.
Now, if I can only get Quicklisp to work again with :DEFSYSTEM-DEPENDS-ON clauses, I will be a pig in slop heaven…
I'm not sure what this issue is, but it's complicated by the fact that QL operates with such an old version of ASDF. In effect this means that QL will operate with whatever is the current ASDF in your environment, because ASDF won't downgrade itself. This means that people are running with all kinds of different ASDF versions, making debugging difficult.
Thanks very much for the help.
No problem!
Best, r
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
-
Faré -
Mark Evenson -
Robert Goldman