[asdf-devel] TEST-OP made useful by declarative options
The attached patch adds a new option to DEFSYSTEM. This option is :TESTS test-list test-list = (:system system-name) | function-name system-name = keyword | string function-name = symbol | string The function name is a string designator that denotes a function with no arguments to be called. This string will be read using the standard I/O syntax only when the test is executed. This ensures that the mechanism works once the appropriate dependencies are loaded. The system name denotes a system that will be loaded when the TEST-OP operation is invoked. The patch can be improved with some tests done when executing DEFSYTEM instead of what it does now, which is to signal an error once the test is performed. The usual example follows. foo.asd: (defsystem :foo :tests ((:system "faa")) :components ((:file "foo"))) faa.asd: (defsystem :faa :tests ("faa::run-tests") :components ((:file "faa"))) foo.lisp: (defpackage "FOO") (in-package "FOO") (defun my-cos (a) (cos a)) faa.lisp: (defpackage "FAA") (in-package "FAA") (defun run-tests () (format t "~&;;;~%;;; MY-COS signals errors: ~A~%;;;" (handler-case (progn (foo::my-cos 'a) 'no) (error (c) 'yes)))) -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
Forgot the diff file. -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
On 4/17/10 Apr 17 -2:18 PM, Juan Jose Garcia-Ripoll wrote:
The attached patch adds a new option to DEFSYSTEM. This option is
:TESTS test-list
test-list = (:system system-name) | function-name system-name = keyword | string function-name = symbol | string
The function name is a string designator that denotes a function with no arguments to be called. This string will be read using the standard I/O syntax only when the test is executed. This ensures that the mechanism works once the appropriate dependencies are loaded. The system name denotes a system that will be loaded when the TEST-OP operation is invoked.
The patch can be improved with some tests done when executing DEFSYTEM instead of what it does now, which is to signal an error once the test is performed.
I applaud the idea of having a declarative spec of tests to be run, but is the test function really the only way to do this? Seems like there are test frameworks where what you want to do is to specify (somehow) a set of test objects to be run, where the framework "knows" how to run them. suggestion: if we are to propose an ASDF extension for testing, we should first survey the CL test frameworks (off-hand: FiveAM, NST, cl-unit, LIFT, RT, and ? ...) and see what they would most "like" in the way of a test specification. Then we can decide how to extend ASDF to suit. Alternatively, possibly different test frameworks could propose different extensions. (defsystem foo :lift-tests ... ) etc. Best, r
On Sat, Apr 17, 2010 at 10:29 PM, Robert Goldman <rpgoldman@sift.info>wrote:
I applaud the idea of having a declarative spec of tests to be run, but is the test function really the only way to do this? Seems like there are test frameworks where what you want to do is to specify (somehow) a set of test objects to be run, where the framework "knows" how to run them.
There are several reasons why I used the function name and did not create framework specific options * People are using their own custom functions for running tests, and never in an uniform way. * It was argued here that ASDF should never be bound to any particular test framework. I have become convinced of the arguments in that respect. * We can not impose test frameworks how to behave, but we can recommend users what they should output their tests (please have a look at the thread I started long ago).
suggestion: if we are to propose an ASDF extension for testing, we should first survey the CL test frameworks (off-hand: FiveAM, NST, cl-unit, LIFT, RT, and ? ...) and see what they would most "like" in the way of a test specification.
Given that the :tests keyword work, I have a mean to do what you want without really binding ASDF to the test framework. The idea would be to provide plug-ins for those frameworks with system classes that take those new options. We would then extend the syntax of :tests to include closures (would be great!) and let the system-test class set the appropriate value of :tests. But note that is completely independent of the current patch and does not preclude or make it less useful, since users may also want to write their own tests. But what this does bring into the light is the need of means to describe that a given DEFSYSTEM form depends on a given plug in or other system. This would be what I was calling the :ASDF-DEPENDENCIES or :ASDF-SUPPORT option in previous emails. Something like (defsystem :my-test-system :asdf-dependencies (:rt-system) :rt-tests ("TEST1" "TEST2" ...) ...) Let's see if I can produce something like that tonight -- stranded in Madrid because of the volcano cloud :-/ Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
A simple example that takes the RT package, which is a rather fragile testing framework, and wraps it inside an ASDF system using the previous ideas I just coded. This allows declaring the dependency on the given plug-in, using the new class and relying on the fact that the tests will be automagically recorded in the system. How this works? - The new test class, specialized for RT, wraps new components which are "actions" before and at the end of any LOAD process. - These actions save and restore the values of the variables in the RT package which are responsible for the tests being run. - When the test system is loaded, RT stores the tests in the variables and the last action saves them all in the ASDF system. - When the test system is first used, the class prepares a new component in the SYSTEM-TESTS fields which is a closure. The plug-in in asdf-rt.lisp reveals various deficiencies: - If we want a purely declarative syntax, without side effects, we have to specify symbols in other packages using strings (see the asdf-rt:rt-test-system) clas below. - The parsing of DEFSYSTEM currently can not be customized by the class. - Adding components to a module is not done using SHARED-INITIALIZE and this is bad because, once more, additional classes can not extend or process the list of components. CL-USER> (asdf:test-system :my-package) [...] ; loading system definition from ; /Users/jjgarcia/devel/asdf/my-package-test.asd into #<PACKAGE "ASDF0"> ; registering #<RT-TEST-SYSTEM :MY-PACKAGE-TEST {1281F7A1}> as ; MY-PACKAGE-TEST Doing 2 pending tests of 2 tests total. Test MY-PACKAGE-TEST::TEST-001 failed Form: (MY-FUNCTION 0) Expected value: 1.0 Actual value: #<UNDEFINED-FUNCTION MY-FUNCTION {120504E9}>. Test MY-PACKAGE-TEST::TEST-002 failed Form: (HANDLER-CASE (AND (MY-FUNCTION 'MY-PACKAGE-TEST::A) NIL) (ERROR (MY-PACKAGE-TEST::C) T)) Expected value: NIL Actual value: T. 2 out of 2 total tests failed: MY-PACKAGE-TEST::TEST-001, my-package.asd: (defsystem :my-package :components ((:file "my-package")) :tests ((:system :my-package-test))) my-package.lisp: (in-package :cl-user) (defun my-function (a) (cos a)) my-package-test.asd: (defsystem :my-package-test :asdf-dependencies (:asdf-rt) :class "ASDF-RT:RT-TEST-SYSTEM" :serial t :components ((:file "my-package-test"))) my-package-test.lisp: (defpackage :my-package-test (:use #+sbcl :sb-rt #+ecl :rt :cl)) (in-package :my-package-test) (deftest test-001 (cl-user::my-function 0) 1.0) (deftest test-002 (handler-case (and (cl-user::my-function 'a) nil) (error (c) t)) nil) -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
On 4/17/10 Apr 17 -6:10 PM, Juan Jose Garcia-Ripoll wrote:
A simple example that takes the RT package, which is a rather fragile testing framework, and wraps it inside an ASDF system using the previous ideas I just coded.
....
The plug-in in asdf-rt.lisp reveals various deficiencies:
- If we want a purely declarative syntax, without side effects, we have to specify symbols in other packages using strings (see the asdf-rt:rt-test-system) clas below. - The parsing of DEFSYSTEM currently can not be customized by the class.
With all due respect, this is at one and the same time not true, and the way it should be(!) Not true: DEFSYSTEM is a macro which means that any ASDF extension can call for arbitrary s-expressions to be passed as argument values, and can then process these arguments to its own satisfaction. It's a feature: 1. I am surprised to hear you, of all people, call for allowing ASDF extensions to customize the parsing of DEFSYSTEM, since this seems to directly contradict your desire for ASDF to become more declarative. 2. The current arrangement permits extended ASDF systems to be processed by core ASDF. For example, if a user doesn't have the test framework (and I don't believe that we should ever force an ASDF library user to install a particular test library), ASDF will simply ignore initargs that are appropriate only to the test library, because of DEFSYSTEM's permissive &allow-other-keys argument processing.
- Adding components to a module is not done using SHARED-INITIALIZE and this is bad because, once more, additional classes can not extend or process the list of components.
This is an interesting point. What obstacles would there be to revising the defsystem processor to use SHARED-INITIALIZE in ASDF 2+? Or, if SHARED-INITIALIZE would be inappropriate, could we add an ASDF API method that would do something similar, and which would allow for user configuration of an initialization protocol. Best, r
On Sun, Apr 18, 2010 at 4:32 PM, Robert Goldman <rpgoldman@sift.info> wrote:
On 4/17/10 Apr 17 -6:10 PM, Juan Jose Garcia-Ripoll wrote:
- The parsing of DEFSYSTEM currently can not be customized by the class.
With all due respect, this is at one and the same time not true, and the way it should be(!) [...] 1. I am surprised to hear you, of all people, call for allowing ASDF extensions to customize the parsing of DEFSYSTEM, since this seems to directly contradict your desire for ASDF to become more declarative
I realized that, as usual, I did not express myself properly. I should have not split the two points in DEFSYSTEM and SHARED-INITIALIZE. My actual complain was that there is no way to add options to a class and let the class process the system definition based on them. Not really the parsing: if DEFSYSTEM is implemented using SHARED-INITIALIZE and the class gets the original form, then it could implement additional parsing steps, such as ensuring that there is a configuration file for lift, or placing some safety actions before and at the end of the test system. I am not at all for classes changing the DEFSYSTEM grammar, or introducing hidden dependencies themselves. That would go against any effort on turning system definitions into descriptive files that can be processed independently of any configuration one may have. Thanks a lot for clarifying this point. Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
participants (2)
-
Juan Jose Garcia-Ripoll -
Robert Goldman