Re: PATCH: SIGNAL a condition on test failures, for use with ASDF:TEST-SYSTEM
Vladimir -- maybe compare this suggestion with what is in the fiveam-asdf contrib in the ASDF repo? Also (and we should probably move this to ASDF-DEVEL) it would be good to have *all* of the tests run to completion even if one or more fail by default, and have the option to exit after the first failure. This could be done by having an `ASDF-SINGLE-TEST-FAILURE` and an `ASDF-TEST-FAILURE` and raising the first at each failure, but by default trapping them and bundling them into the `ASDF-TEST-FAILURE` condition. It might make sense to bundle FiveAM's test report *into* the `ASDF-TEST-FAILURE` (or subclass thereof), so that it can be printed at an appropriate time. On 14 Sep 2019, at 20:48, Vladimir Sedach wrote:
Hello!
I have a proposed patch to FiveAM on GitHub: https://github.com/sionescu/fiveam/pull/58
I recently wrote a script to run FiveAM tests for one of my libraries on many different implementations on your own machine: https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-te...
I do not want to copy-paste that script for all of my libraries. It would be nice to contribute a generalized version of the script to Roswell (on which the script is based), and just be able to say "ros test my-system in all installed implementations" for any system.
The first step to doing that is to get ASDF:TEST-SYSTEM to report errors using a common interface.
Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use conditions to signal test failures. I like this idea. Since you do not have to handle conditions raised by SIGNAL, the functionality can be added without impacting existing use or interfaces of FiveAM (the use of WARNING instead of ERROR as the parent condition of TEST-SPEC-FAILURE also ensures this, as many people tend to abuse HANDLER-CASE to indiscriminately catch ERRORs).
With this patch, all projects that use FiveAM and define ASDF test-op should be testable from a generic version of the Roswell script without any changes.
The basic mechanism for test automation with FiveAM is then very simple:
(handler-case (asdf:test-system "system") (fiveam:test-spec-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
Other test libraries can have supported added as well:
(handler-case (asdf:test-system "system") ((or fiveam:test-spec-failure prove:test-failure) (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
The goal is to introduce a parent condition in ASDF that FiveAM, etc., will then inherit from:
(handler-case (asdf:test-system "system") (asdf:test-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
Ultimately, any system using any test library should be testable this way, ensuring that test automation becomes trivial to build.
I am going to propose adding the parent condition to ASDF, but in the meantime (until the ASDF patch makes it to the repository, another release comes out, release becomes widely available…), I would like to get this code into FiveAM.
Vladimir
Note that my recommended solution is for a test library to create a report file with a test-report-op (that always succeeds at creating a report file, but the report file may indicate failure), and the test-op just depends on the test-report-op, and signals and error if it didn't indicate success. For extra win, the test system may even use fine-grained dependencies, wherein each file in the test system would depend on loading a specific file (or set of files) in the tested system rather than on loading the entire system; then it would not recompute a test if its dependencies have not changed—but that probably wouldn't be safe unless these dependencies are actually enforced like xcvb or bazelisp would (i.e. uiop wouldn't do). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org #.(let((q`(#0=lambda(q)(labels((p(q)(if(atom q)(intern(#1=reverse(string q)))(#1#(mapcar #'p q))))(q(q)(subst q(eq q q)'(#2=defun p(&aux #2#nufed ,@''#3=etouq(xua& #3#)p tsil)((#0#(q)(setq q 't tsil q nufed(eval(list q (list'quote q)))))#3#)))))(nconc(q q)(p(q q)))))))(eval`(,q',q))) On Sat, Sep 14, 2019 at 9:58 PM Robert Goldman <rpgoldman@sift.info> wrote:
Vladimir -- maybe compare this suggestion with what is in the fiveam-asdf contrib in the ASDF repo?
Also (and we should probably move this to ASDF-DEVEL) it would be good to have all of the tests run to completion even if one or more fail by default, and have the option to exit after the first failure.
This could be done by having an ASDF-SINGLE-TEST-FAILURE and an ASDF-TEST-FAILURE and raising the first at each failure, but by default trapping them and bundling them into the ASDF-TEST-FAILURE condition.
It might make sense to bundle FiveAM's test report into the ASDF-TEST-FAILURE (or subclass thereof), so that it can be printed at an appropriate time.
On 14 Sep 2019, at 20:48, Vladimir Sedach wrote:
Hello!
I have a proposed patch to FiveAM on GitHub: https://github.com/sionescu/fiveam/pull/58
I recently wrote a script to run FiveAM tests for one of my libraries on many different implementations on your own machine: https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-te...
I do not want to copy-paste that script for all of my libraries. It would be nice to contribute a generalized version of the script to Roswell (on which the script is based), and just be able to say "ros test my-system in all installed implementations" for any system.
The first step to doing that is to get ASDF:TEST-SYSTEM to report errors using a common interface.
Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use conditions to signal test failures. I like this idea. Since you do not have to handle conditions raised by SIGNAL, the functionality can be added without impacting existing use or interfaces of FiveAM (the use of WARNING instead of ERROR as the parent condition of TEST-SPEC-FAILURE also ensures this, as many people tend to abuse HANDLER-CASE to indiscriminately catch ERRORs).
With this patch, all projects that use FiveAM and define ASDF test-op should be testable from a generic version of the Roswell script without any changes.
The basic mechanism for test automation with FiveAM is then very simple:
(handler-case (asdf:test-system "system") (fiveam:test-spec-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
Other test libraries can have supported added as well:
(handler-case (asdf:test-system "system") ((or fiveam:test-spec-failure prove:test-failure) (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
The goal is to introduce a parent condition in ASDF that FiveAM, etc., will then inherit from:
(handler-case (asdf:test-system "system") (asdf:test-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
Ultimately, any system using any test library should be testable this way, ensuring that test automation becomes trivial to build.
I am going to propose adding the parent condition to ASDF, but in the meantime (until the ASDF patch makes it to the repository, another release comes out, release becomes widely available…), I would like to get this code into FiveAM.
Vladimir
Faré <fahree@gmail.com> writes:
Note that my recommended solution is for a test library to create a report file with a test-report-op (that always succeeds at creating a report file, but the report file may indicate failure), and the test-op just depends on the test-report-op, and signals and error if it didn't indicate success.
What branch of the ASDF repository is this in? These are the only two mentions of test-report-op in master: TODO:23:** Replace test-op with run-test-op and test-report-op ? doc/asdf.texinfo:2743:on a deterministically repeatable @code{test-report-op}, In particular this paragraph: Of course, the @code{test-op} for your system could depend on a deterministically repeatable @code{test-report-op}, and just read the results from the report files, in which case you could have this method return @code{t}. I do not understand how test-report-op would help in making a generic version of the test script I posted: https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-te... Vladimir
Note that my recommended solution is for a test library to create a report file with a test-report-op (that always succeeds at creating a report file, but the report file may indicate failure), and the test-op just depends on the test-report-op, and signals and error if it didn't indicate success.
What branch of the ASDF repository is this in? These are the only two mentions of test-report-op in master:
There is no branch of ASDF doing this, and so far as I know you don't need one: just define your own subclass of system for which test-op has a suitable method. And if you're using my advanced proposal and the inherited methods don't suit you, just skip the (call-next-method).
I do not understand how test-report-op would help in making a generic version of the test script I posted: https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-te...
The two are orthogonal. What test-report-op would do is help avoid re-testing systems that haven't changed, if and when you do massive incremental testing of lots of systems. Your script looks mostly good. I would use lisp-invocation instead of ros, but I can see merit to ros, too. —♯ƒ • François-René Rideau The state is the coldest of all cold monsters. Coldly it lies, too; and this lie creeps from its mouth: "I, the state, am the people." — Nietzsche
Robert Goldman <rpgoldman@sift.info> writes:
Vladimir -- maybe compare this suggestion with what is in the fiveam-asdf contrib in the ASDF repo?
Yes, I looked at fiveam-asdf before doing this. I think the two have very different goals: fiveam-asdf is a comprehensive integration of FiveAM into ASDF, while what I am proposing is a minimal signal protocol that is easy to add to any testing library.
This could be done by having an `ASDF-SINGLE-TEST-FAILURE` and an `ASDF-TEST-FAILURE` and raising the first at each failure, but by default trapping them and bundling them into the `ASDF-TEST-FAILURE` condition.
What is the use case for this? It would be a lot more complicated for test libraries to implement, and I cannot think of any scenarios where I would need that.
It might make sense to bundle FiveAM's test report *into* the `ASDF-TEST-FAILURE` (or subclass thereof), so that it can be printed at an appropriate time.
Yes, that is what my patch to FiveAM does: https://github.com/sionescu/fiveam/pull/58/files#diff-176b9e23e37825f8c99733... Vladimir
Hi Vladimir, The idea of having a protocol between ASDF and test frameworks is very good, but I don't think this is enough. I'd like to have something that gathers enough information to allow building a suitable integration with continuous integration systems (there are already a few ad-hoc ones around). For example, a common taxonomy of test status is: SUCEEDED, FAILED, BROKEN, SKIPPED, NOTSTARTED. Whatever support ends up in ASDF should be sufficient to allow outputting common rest report formats, such as JUnit, TAP, etc... otherwise we risk ending up with something that cannot be improved because users start relying on it and we don't want to break backwards compatibility. Mainly for this reason I also won't merge the FiveAM pull request. Until the ASDF support is in, adding some support to FiveAM is premature.
This could be done by having an `ASDF-SINGLE-TEST-FAILURE` and an `ASDF-TEST-FAILURE` and raising the first at each failure, but by default trapping them and bundling them into the `ASDF-TEST-FAILURE` condition.
It might make sense to bundle FiveAM's test report *into* the `ASDF-TEST-FAILURE` (or subclass thereof), so that it can be printed at an appropriate time.
On 14 Sep 2019, at 20:48, Vladimir Sedach wrote:
Hello!
I have a proposed patch to FiveAM on GitHub: https://github.com/sionescu/fiveam/pull/58
I recently wrote a script to run FiveAM tests for one of my libraries on many different implementations on your own machine: https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-te...
I do not want to copy-paste that script for all of my libraries. It would be nice to contribute a generalized version of the script to Roswell (on which the script is based), and just be able to say "ros test my-system in all installed implementations" for any system.
The first step to doing that is to get ASDF:TEST-SYSTEM to report errors using a common interface.
Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use conditions to signal test failures. I like this idea. Since you do not have to handle conditions raised by SIGNAL, the functionality can be added without impacting existing use or interfaces of FiveAM (the use of WARNING instead of ERROR as the parent condition of TEST-SPEC-FAILURE also ensures this, as many people tend to abuse HANDLER-CASE to indiscriminately catch ERRORs).
With this patch, all projects that use FiveAM and define ASDF test-op should be testable from a generic version of the Roswell script without any changes.
The basic mechanism for test automation with FiveAM is then very simple:
(handler-case (asdf:test-system "system") (fiveam:test-spec-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
Other test libraries can have supported added as well:
(handler-case (asdf:test-system "system") ((or fiveam:test-spec-failure prove:test-failure) (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
The goal is to introduce a parent condition in ASDF that FiveAM, etc., will then inherit from:
(handler-case (asdf:test-system "system") (asdf:test-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
Ultimately, any system using any test library should be testable this way, ensuring that test automation becomes trivial to build.
I am going to propose adding the parent condition to ASDF, but in the meantime (until the ASDF patch makes it to the repository, another release comes out, release becomes widely available…), I would like to get this code into FiveAM.
Vladimir
-- Stelian Ionescu
participants (4)
-
Faré -
Robert Goldman -
Stelian Ionescu -
Vladimir Sedach