September 2019 - asdf-devel - mailman3.common-lisp.net

[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class
by Vladimir Sedach 26 Mar '21

26 Mar '21

Hello! I recently wrote a script to run FiveAM tests for one of my libraries on many different implementations on your own machine: https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-t… It would be really nice if I did not have to copy-paste that script to my other libraries, and instead could contribute a generalized version to Roswell (on which the script is based) and have it work for any project using any test library. What I would like to be able to do, for any system: (handler-case (asdf:test-system "system") (asdf:test-op-test-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1))) The attached patch adds an ASDF:TEST-OP-TEST-FAILURE condition that test libraries can inherit from. I also added the necessary functionality to FiveAM: https://github.com/sionescu/fiveam/pull/58 It should be easy to add similar functionality to other testing libraries. This will make test automation trivial, with few, if any, changes required to systems that use ASDF and the testing libraries. One thing I would like to discuss is which condition ASDF:TEST-OP-TEST-FAILURE should inherit from. It definitely should not be ERROR - there is no error in test-op, or in running the tests; test failures are a regular and expected occurrence of running tests. Also problematic is widespread abuse of HANDLER-CASE to catch any ERROR; I am afraid if signaled from popular test libraries it would break someone's test running code somewhere. WARNING seems like a nice parent condition, but maybe having ASDF:TEST-OP-TEST-FAILURE inherit from CONDITION is a better idea. Thoughts? Vladimir

4 15

test-op and dependencies
by Anton Vodonosov 01 Oct '19

01 Oct '19

How test-op is expected to deal with tests in dependency libraries? If my application depends on several libraries, which in turn depend on others, when a user of my application asdf system runs test-op, is it expected to only run tests of the application itself? Of my test-op implementation should explicitly load and call tests of all dependencies? I think it's better to recommend test-op implementations to run only tests of this system itself and ASDF could provide a "recursive test op", which runs tests on the whole dependency tree.

2 1

Re: PATCH: SIGNAL a condition on test failures, for use with ASDF:TEST-SYSTEM
by Robert Goldman 16 Sep '19

16 Sep '19

Vladimir -- maybe compare this suggestion with what is in the fiveam-asdf contrib in the ASDF repo? Also (and we should probably move this to ASDF-DEVEL) it would be good to have *all* of the tests run to completion even if one or more fail by default, and have the option to exit after the first failure. This could be done by having an `ASDF-SINGLE-TEST-FAILURE` and an `ASDF-TEST-FAILURE` and raising the first at each failure, but by default trapping them and bundling them into the `ASDF-TEST-FAILURE` condition. It might make sense to bundle FiveAM's test report *into* the `ASDF-TEST-FAILURE` (or subclass thereof), so that it can be printed at an appropriate time. On 14 Sep 2019, at 20:48, Vladimir Sedach wrote: > Hello! > > I have a proposed patch to FiveAM on GitHub: > https://github.com/sionescu/fiveam/pull/58 > > I recently wrote a script to run FiveAM tests for one of my > libraries on many different implementations on your own machine: > https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-t… > > I do not want to copy-paste that script for all of my libraries. It > would be nice to contribute a generalized version of the script to > Roswell (on which the script is based), and just be able to say "ros > test my-system in all installed implementations" for any system. > > The first step to doing that is to get ASDF:TEST-SYSTEM to report > errors using a common interface. > > Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use > conditions to signal test failures. I like this idea. Since you do > not have to handle conditions raised by SIGNAL, the functionality can > be added without impacting existing use or interfaces of FiveAM (the > use of WARNING instead of ERROR as the parent condition of > TEST-SPEC-FAILURE also ensures this, as many people tend to abuse > HANDLER-CASE to indiscriminately catch ERRORs). > > With this patch, all projects that use FiveAM and define ASDF test-op > should be testable from a generic version of the Roswell script > without any changes. > > The basic mechanism for test automation with FiveAM is then very > simple: > > (handler-case (asdf:test-system "system") > (fiveam:test-spec-failure (condition) > (princ condition uiop:*stderr*) > (uiop:quit 1))) > > Other test libraries can have supported added as well: > > (handler-case (asdf:test-system "system") > ((or fiveam:test-spec-failure prove:test-failure) (condition) > (princ condition uiop:*stderr*) > (uiop:quit 1))) > > The goal is to introduce a parent condition in ASDF that FiveAM, etc., > will then inherit from: > > (handler-case (asdf:test-system "system") > (asdf:test-failure (condition) > (princ condition uiop:*stderr*) > (uiop:quit 1))) > > Ultimately, any system using any test library should be testable this > way, ensuring that test automation becomes trivial to build. > > I am going to propose adding the parent condition to ASDF, but in the > meantime (until the ASDF patch makes it to the repository, another > release comes out, release becomes widely available…), I would like > to get this code into FiveAM. > > Vladimir

4 5

trouble with upgrade of asdf and versions
by adam 06 Sep '19

06 Sep '19

In attempting to upgrade ASDF, I have broken its operation. This work-around has not been helpful. https://stackoverflow.com/q/45043190/8770050 Here is the problem, (and Quicklisp seems to be working OK) CL-USER> (ql:quickload :quickproject) ; Evaluation aborted on #<ASDF-2.26:LOAD-SYSTEM-DEFINITION-ERROR #x213E199E>. CL-USER> (asdf:asdf-version) "3.3.2" CL-USER> *features* (:SWANK :ASDF3.3 :ASDF3.2 :ASDF3.1 :ASDF3 :OS-UNIX :QUICKLISP :ASDF2 :ASDF-UNICODE :CM :SAL :METACLASSES :ASDF :READLINE :REGEXP :SYSCALLS :I18N :LOOP :COMPILER :CLOS :MOP :CLISP :ANSI-CL :COMMON-LISP :LISP=CL :INTERPRETER :SOCKETS :GENERIC-STREAMS :LOGICAL-PATHNAMES :SCREEN :FFI :GETTEXT :UNICODE :BASE-CHAR=CHARACTER :PC386 :UNIX) My Lisp and Emacs versions, on Ubuntu 16.04, are $ clisp --version GNU CLISP 2.49 (2010-07-07) (built on allspice.buildd [127.0.1.1]) Emacs version GNU Emacs 24.5.1 (i686-pc-linux-gnu, GTK+ Version 3.18.9) of 2017-09-21 on lgw01-amd64- 050, modified by Debian Any suggestions on repairing or upgrading my ASDF, gratefully received.

5 9