Hi all,
Right now I'm running unit tests using Fukamachi's prove library: (asdf/operate:test-system 'my-system).
I want to run my tests generating the tests coverage of my system.
For this I would like to have similar operation, but which will: 1) turn on the code coverage in LispWorks (just call to some global function) 2) rebuild system which I want to test (in this case my-system), but only (!) this system (not dependencies) 3) run tests 4) call coverage results processing function. 5) on normal run (asdf/operate:test-system 'my-system) rebuild the system without coverage.
How could I proceed with this task?
Anyone ? Is impossible to do or too hard? I see it as following: make a new operation which will: 1) compile the test cases 1) clean the test object (remove fasls of target system to test but not dependencies) 2) execute necessary startup code (set up the code coverage) 3) run testcases 4) perform teardown code (turn off the code coverage and collect stats etc)
How could I achieve at least this?
Alexey Veretennikov txm.fourier@gmail.com writes:
Hi all,
Right now I'm running unit tests using Fukamachi's prove library: (asdf/operate:test-system 'my-system).
I want to run my tests generating the tests coverage of my system.
For this I would like to have similar operation, but which will:
- turn on the code coverage in LispWorks (just call to some global
function) 2) rebuild system which I want to test (in this case my-system), but only (!) this system (not dependencies) 3) run tests 4) call coverage results processing function. 5) on normal run (asdf/operate:test-system 'my-system) rebuild the system without coverage.
How could I proceed with this task?
I recommend that you write run tests, etc., in a separate process, as orchestrated by a script that just after it loads ASDF 1- loads all the library code for which you do NOT want test coverage 2- turns on coverage 3- configures the asdf-output-translations to redirect object files for those systems that you DO want coverage (and only those) to an alternate location 4- load the rest of the code 5- runs the test
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Reasons for existence are usually provided for things that don't exist; they would be wasted on things which do. — Saul Gorn
On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Anyone ? Is impossible to do or too hard? I see it as following: make a new operation which will:
- compile the test cases
- clean the test object (remove fasls of target system to test but not
dependencies) 2) execute necessary startup code (set up the code coverage) 3) run testcases 4) perform teardown code (turn off the code coverage and collect stats etc)
How could I achieve at least this?
Alexey Veretennikov txm.fourier@gmail.com writes:
Hi all,
Right now I'm running unit tests using Fukamachi's prove library: (asdf/operate:test-system 'my-system).
I want to run my tests generating the tests coverage of my system.
For this I would like to have similar operation, but which will:
- turn on the code coverage in LispWorks (just call to some global
function) 2) rebuild system which I want to test (in this case my-system), but only (!) this system (not dependencies) 3) run tests 4) call coverage results processing function. 5) on normal run (asdf/operate:test-system 'my-system) rebuild the system without coverage.
How could I proceed with this task?
-- Br, /Alexey
Hi,
I've done something slightly more brute-force and probably not elegant. I've modified the test asd file for my library in the following way (see below)
Basically I've tried to introduce the same operation as test-op. This helps me to keep dependencies loaded/compiled via quicklisp. But it will 1) Turn on the coverage 2) Force reload of test subject with load-op and :force t 3) Run tests 4) Dump coverage results and [optionally] open browser with them.
Please let me know if it could be done in an easier/more elegant way or if there are any drawbacks of this approach.
#| Test package. Usage: (ql:quickload :mylib-api-test) (asdf/operate:test-system :mylib-api)
In order to perform code coverage with these tests (currenty supported only on LispWorks 7), run the following: (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test) |#
(in-package :cl-user) (defpackage mylib-api-test-asd (:use :cl :asdf) (:export coverage-op))
(in-package :mylib-api-test-asd)
(defclass coverage-op (selfward-operation) ((selfward-operation :initform 'load-op :allocation :class)) (:documentation "Test coverage operation"))
(defsystem mylib-api-test :author "Alexey Veretennikov" :license "BSD" :depends-on (:mylib-api :cl-fad :flexi-streams :prove) :components ((:module "t" :components ((:file "base") (:test-file "pack-test") (:test-file "utils-test")))) :description "Test system for mylib-api"
:defsystem-depends-on (:prove-asdf) :perform (test-op :after (op c) (funcall (intern #.(string :run-test-system) :prove-asdf) c) (asdf:clear-system c)) :perform (coverage-op (op c) (run-tests-with-coverage) (asdf:clear-system c)))
#+lispworks7 (defun generate-coverage-output-path () (multiple-value-bind (second minute hour date month year);; day) (get-decoded-time) (let ((results-directory-name (pathname (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html" year month date hour minute second))))
(merge-pathnames results-directory-name (hcl:get-temp-directory)))))
#+lispworks7 (defun run-lw-test-coverage () (hcl:clear-code-coverage) (hcl:with-code-coverage-generation () (asdf/operate:load-system :mylib-api :force t)) (asdf/operate:test-system :mylib-api-test) (let ((output-file (generate-coverage-output-path))) (hcl:code-coverage-data-generate-coloring-html output-file) (format *standard-output* "Generated coverage report to ~a" output-file) #+macosx (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:" (objc:invoke "NSURL" "URLWithString:" (concatenate 'string "file://" (namestring output-file))))))
(defun run-tests-with-coverage () #+lispworks7 (run-lw-test-coverage) #-lispworks7 (error "Code coverage generation currently supported only on LispWorks 7 and above"))
Faré fahree@gmail.com writes:
I recommend that you write run tests, etc., in a separate process, as orchestrated by a script that just after it loads ASDF 1- loads all the library code for which you do NOT want test coverage 2- turns on coverage 3- configures the asdf-output-translations to redirect object files for those systems that you DO want coverage (and only those) to an alternate location 4- load the rest of the code 5- runs the test
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Reasons for existence are usually provided for things that don't exist; they would be wasted on things which do. — Saul Gorn
On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Anyone ? Is impossible to do or too hard? I see it as following: make a new operation which will:
- compile the test cases
- clean the test object (remove fasls of target system to test but not
dependencies) 2) execute necessary startup code (set up the code coverage) 3) run testcases 4) perform teardown code (turn off the code coverage and collect stats etc)
How could I achieve at least this?
Alexey Veretennikov txm.fourier@gmail.com writes:
Hi all,
Right now I'm running unit tests using Fukamachi's prove library: (asdf/operate:test-system 'my-system).
I want to run my tests generating the tests coverage of my system.
For this I would like to have similar operation, but which will:
- turn on the code coverage in LispWorks (just call to some global
function) 2) rebuild system which I want to test (in this case my-system), but only (!) this system (not dependencies) 3) run tests 4) call coverage results processing function. 5) on normal run (asdf/operate:test-system 'my-system) rebuild the system without coverage.
How could I proceed with this task?
-- Br, /Alexey
1- Calling clear-system from within perform is wrong. 2- I still recommend redirecting the output-translations for that particular tree around the load-system with coverage, and dropping the :force t.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "I believe that sex is one of the most beautiful, natural, wholesome things that money can buy." — Steve Martin
On Tue, Oct 18, 2016 at 6:21 PM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Hi,
I've done something slightly more brute-force and probably not elegant. I've modified the test asd file for my library in the following way (see below)
Basically I've tried to introduce the same operation as test-op. This helps me to keep dependencies loaded/compiled via quicklisp. But it will
- Turn on the coverage
- Force reload of test subject with load-op and :force t
- Run tests
- Dump coverage results and [optionally] open browser with them.
Please let me know if it could be done in an easier/more elegant way or if there are any drawbacks of this approach.
#| Test package. Usage: (ql:quickload :mylib-api-test) (asdf/operate:test-system :mylib-api)
In order to perform code coverage with these tests (currenty supported only on LispWorks 7), run the following: (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test) |#
(in-package :cl-user) (defpackage mylib-api-test-asd (:use :cl :asdf) (:export coverage-op))
(in-package :mylib-api-test-asd)
(defclass coverage-op (selfward-operation) ((selfward-operation :initform 'load-op :allocation :class)) (:documentation "Test coverage operation"))
(defsystem mylib-api-test :author "Alexey Veretennikov" :license "BSD" :depends-on (:mylib-api :cl-fad :flexi-streams :prove) :components ((:module "t" :components ((:file "base") (:test-file "pack-test") (:test-file "utils-test")))) :description "Test system for mylib-api"
:defsystem-depends-on (:prove-asdf) :perform (test-op :after (op c) (funcall (intern #.(string :run-test-system) :prove-asdf) c) (asdf:clear-system c)) :perform (coverage-op (op c) (run-tests-with-coverage) (asdf:clear-system c)))
#+lispworks7 (defun generate-coverage-output-path () (multiple-value-bind (second minute hour date month year);; day) (get-decoded-time) (let ((results-directory-name (pathname (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html" year month date hour minute second))))
(merge-pathnames results-directory-name (hcl:get-temp-directory)))))#+lispworks7 (defun run-lw-test-coverage () (hcl:clear-code-coverage) (hcl:with-code-coverage-generation () (asdf/operate:load-system :mylib-api :force t)) (asdf/operate:test-system :mylib-api-test) (let ((output-file (generate-coverage-output-path))) (hcl:code-coverage-data-generate-coloring-html output-file) (format *standard-output* "Generated coverage report to ~a" output-file) #+macosx (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:" (objc:invoke "NSURL" "URLWithString:" (concatenate 'string "file://" (namestring output-file))))))
(defun run-tests-with-coverage () #+lispworks7 (run-lw-test-coverage) #-lispworks7 (error "Code coverage generation currently supported only on LispWorks 7 and above"))
Faré fahree@gmail.com writes:
I recommend that you write run tests, etc., in a separate process, as orchestrated by a script that just after it loads ASDF 1- loads all the library code for which you do NOT want test coverage 2- turns on coverage 3- configures the asdf-output-translations to redirect object files for those systems that you DO want coverage (and only those) to an alternate location 4- load the rest of the code 5- runs the test
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Reasons for existence are usually provided for things that don't exist; they would be wasted on things which do. — Saul Gorn
On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Anyone ? Is impossible to do or too hard? I see it as following: make a new operation which will:
- compile the test cases
- clean the test object (remove fasls of target system to test but not
dependencies) 2) execute necessary startup code (set up the code coverage) 3) run testcases 4) perform teardown code (turn off the code coverage and collect stats etc)
How could I achieve at least this?
Alexey Veretennikov txm.fourier@gmail.com writes:
Hi all,
Right now I'm running unit tests using Fukamachi's prove library: (asdf/operate:test-system 'my-system).
I want to run my tests generating the tests coverage of my system.
For this I would like to have similar operation, but which will:
- turn on the code coverage in LispWorks (just call to some global
function) 2) rebuild system which I want to test (in this case my-system), but only (!) this system (not dependencies) 3) run tests 4) call coverage results processing function. 5) on normal run (asdf/operate:test-system 'my-system) rebuild the system without coverage.
How could I proceed with this task?
-- Br, /Alexey
-- Br, /Alexey
Hi,
I'm trying to understand what you are saying :) Right now I'm having a problem what
(let ((asdf/output-translations:*output-translations* '(((#1=#P"/Users/alexeyv/.cache/common-lisp/lw-7.0.0-macosx-x86/**/profile-*.*" T) (T #1#))))) (asdf/operate:load-system :mylibrary))
would like to recompile all the dependent libraries as well with the new output translations (I'll fix paths later). Is where a way to load exactly this system I want to load, without even trying to load everything which it depends on?
Faré fahree@gmail.com writes:
1- Calling clear-system from within perform is wrong. 2- I still recommend redirecting the output-translations for that particular tree around the load-system with coverage, and dropping the :force t.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "I believe that sex is one of the most beautiful, natural, wholesome things that money can buy." — Steve Martin
On Tue, Oct 18, 2016 at 6:21 PM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Hi,
I've done something slightly more brute-force and probably not elegant. I've modified the test asd file for my library in the following way (see below)
Basically I've tried to introduce the same operation as test-op. This helps me to keep dependencies loaded/compiled via quicklisp. But it will
- Turn on the coverage
- Force reload of test subject with load-op and :force t
- Run tests
- Dump coverage results and [optionally] open browser with them.
Please let me know if it could be done in an easier/more elegant way or if there are any drawbacks of this approach.
#| Test package. Usage: (ql:quickload :mylib-api-test) (asdf/operate:test-system :mylib-api)
In order to perform code coverage with these tests (currenty supported only on LispWorks 7), run the following: (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test) |#
(in-package :cl-user) (defpackage mylib-api-test-asd (:use :cl :asdf) (:export coverage-op))
(in-package :mylib-api-test-asd)
(defclass coverage-op (selfward-operation) ((selfward-operation :initform 'load-op :allocation :class)) (:documentation "Test coverage operation"))
(defsystem mylib-api-test :author "Alexey Veretennikov" :license "BSD" :depends-on (:mylib-api :cl-fad :flexi-streams :prove) :components ((:module "t" :components ((:file "base") (:test-file "pack-test") (:test-file "utils-test")))) :description "Test system for mylib-api"
:defsystem-depends-on (:prove-asdf) :perform (test-op :after (op c) (funcall (intern #.(string :run-test-system) :prove-asdf) c) (asdf:clear-system c)) :perform (coverage-op (op c) (run-tests-with-coverage) (asdf:clear-system c)))
#+lispworks7 (defun generate-coverage-output-path () (multiple-value-bind (second minute hour date month year);; day) (get-decoded-time) (let ((results-directory-name (pathname (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html" year month date hour minute second))))
(merge-pathnames results-directory-name (hcl:get-temp-directory)))))#+lispworks7 (defun run-lw-test-coverage () (hcl:clear-code-coverage) (hcl:with-code-coverage-generation () (asdf/operate:load-system :mylib-api :force t)) (asdf/operate:test-system :mylib-api-test) (let ((output-file (generate-coverage-output-path))) (hcl:code-coverage-data-generate-coloring-html output-file) (format *standard-output* "Generated coverage report to ~a" output-file) #+macosx (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:" (objc:invoke "NSURL" "URLWithString:" (concatenate 'string "file://" (namestring output-file))))))
(defun run-tests-with-coverage () #+lispworks7 (run-lw-test-coverage) #-lispworks7 (error "Code coverage generation currently supported only on LispWorks 7 and above"))
Faré fahree@gmail.com writes:
I recommend that you write run tests, etc., in a separate process, as orchestrated by a script that just after it loads ASDF 1- loads all the library code for which you do NOT want test coverage 2- turns on coverage 3- configures the asdf-output-translations to redirect object files for those systems that you DO want coverage (and only those) to an alternate location 4- load the rest of the code 5- runs the test
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Reasons for existence are usually provided for things that don't exist; they would be wasted on things which do. — Saul Gorn
On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Anyone ? Is impossible to do or too hard? I see it as following: make a new operation which will:
- compile the test cases
- clean the test object (remove fasls of target system to test but not
dependencies) 2) execute necessary startup code (set up the code coverage) 3) run testcases 4) perform teardown code (turn off the code coverage and collect stats etc)
How could I achieve at least this?
Alexey Veretennikov txm.fourier@gmail.com writes:
Hi all,
Right now I'm running unit tests using Fukamachi's prove library: (asdf/operate:test-system 'my-system).
I want to run my tests generating the tests coverage of my system.
For this I would like to have similar operation, but which will:
- turn on the code coverage in LispWorks (just call to some global
function) 2) rebuild system which I want to test (in this case my-system), but only (!) this system (not dependencies) 3) run tests 4) call coverage results processing function. 5) on normal run (asdf/operate:test-system 'my-system) rebuild the system without coverage.
How could I proceed with this task?
-- Br, /Alexey
-- Br, /Alexey
Your output translations rule is non-sensical. Please read the documentation. You need to only translate things under mylibrary/
Actually, redirecting using output-translations will make things work, but is not the right thing to do. The right thing to do is to define operations compile-coverage-op, load-coverage-op, prepare-coverage-op and test-coverage-op that do the right thing, maybe inheriting from compile-op, load-op, prepare-op, test-op (respectively). Then the input-files and output-files for these actions can do the right thing and add a suffix to the file name, etc.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Give up all hope for a better yesterday, even a better just now. Never give up hope for a better tomorrow. — Patri Friedman
On Wed, Oct 19, 2016 at 5:34 PM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Hi,
I'm trying to understand what you are saying :) Right now I'm having a problem what
(let ((asdf/output-translations:*output-translations* '(((#1=#P"/Users/alexeyv/.cache/common-lisp/lw-7.0.0-macosx-x86/**/profile-*.*" T) (T #1#))))) (asdf/operate:load-system :mylibrary))
would like to recompile all the dependent libraries as well with the new output translations (I'll fix paths later). Is where a way to load exactly this system I want to load, without even trying to load everything which it depends on?
Faré fahree@gmail.com writes:
1- Calling clear-system from within perform is wrong. 2- I still recommend redirecting the output-translations for that particular tree around the load-system with coverage, and dropping the :force t.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "I believe that sex is one of the most beautiful, natural, wholesome things that money can buy." — Steve Martin
On Tue, Oct 18, 2016 at 6:21 PM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Hi,
I've done something slightly more brute-force and probably not elegant. I've modified the test asd file for my library in the following way (see below)
Basically I've tried to introduce the same operation as test-op. This helps me to keep dependencies loaded/compiled via quicklisp. But it will
- Turn on the coverage
- Force reload of test subject with load-op and :force t
- Run tests
- Dump coverage results and [optionally] open browser with them.
Please let me know if it could be done in an easier/more elegant way or if there are any drawbacks of this approach.
#| Test package. Usage: (ql:quickload :mylib-api-test) (asdf/operate:test-system :mylib-api)
In order to perform code coverage with these tests (currenty supported only on LispWorks 7), run the following: (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test) |#
(in-package :cl-user) (defpackage mylib-api-test-asd (:use :cl :asdf) (:export coverage-op))
(in-package :mylib-api-test-asd)
(defclass coverage-op (selfward-operation) ((selfward-operation :initform 'load-op :allocation :class)) (:documentation "Test coverage operation"))
(defsystem mylib-api-test :author "Alexey Veretennikov" :license "BSD" :depends-on (:mylib-api :cl-fad :flexi-streams :prove) :components ((:module "t" :components ((:file "base") (:test-file "pack-test") (:test-file "utils-test")))) :description "Test system for mylib-api"
:defsystem-depends-on (:prove-asdf) :perform (test-op :after (op c) (funcall (intern #.(string :run-test-system) :prove-asdf) c) (asdf:clear-system c)) :perform (coverage-op (op c) (run-tests-with-coverage) (asdf:clear-system c)))
#+lispworks7 (defun generate-coverage-output-path () (multiple-value-bind (second minute hour date month year);; day) (get-decoded-time) (let ((results-directory-name (pathname (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html" year month date hour minute second))))
(merge-pathnames results-directory-name (hcl:get-temp-directory)))))#+lispworks7 (defun run-lw-test-coverage () (hcl:clear-code-coverage) (hcl:with-code-coverage-generation () (asdf/operate:load-system :mylib-api :force t)) (asdf/operate:test-system :mylib-api-test) (let ((output-file (generate-coverage-output-path))) (hcl:code-coverage-data-generate-coloring-html output-file) (format *standard-output* "Generated coverage report to ~a" output-file) #+macosx (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:" (objc:invoke "NSURL" "URLWithString:" (concatenate 'string "file://" (namestring output-file))))))
(defun run-tests-with-coverage () #+lispworks7 (run-lw-test-coverage) #-lispworks7 (error "Code coverage generation currently supported only on LispWorks 7 and above"))
Faré fahree@gmail.com writes:
I recommend that you write run tests, etc., in a separate process, as orchestrated by a script that just after it loads ASDF 1- loads all the library code for which you do NOT want test coverage 2- turns on coverage 3- configures the asdf-output-translations to redirect object files for those systems that you DO want coverage (and only those) to an alternate location 4- load the rest of the code 5- runs the test
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Reasons for existence are usually provided for things that don't exist; they would be wasted on things which do. — Saul Gorn
On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov txm.fourier@gmail.com wrote:
Anyone ? Is impossible to do or too hard? I see it as following: make a new operation which will:
- compile the test cases
- clean the test object (remove fasls of target system to test but not
dependencies) 2) execute necessary startup code (set up the code coverage) 3) run testcases 4) perform teardown code (turn off the code coverage and collect stats etc)
How could I achieve at least this?
Alexey Veretennikov txm.fourier@gmail.com writes:
Hi all,
Right now I'm running unit tests using Fukamachi's prove library: (asdf/operate:test-system 'my-system).
I want to run my tests generating the tests coverage of my system.
For this I would like to have similar operation, but which will:
- turn on the code coverage in LispWorks (just call to some global
function) 2) rebuild system which I want to test (in this case my-system), but only (!) this system (not dependencies) 3) run tests 4) call coverage results processing function. 5) on normal run (asdf/operate:test-system 'my-system) rebuild the system without coverage.
How could I proceed with this task?
-- Br, /Alexey
-- Br, /Alexey
-- Br, /Alexey
-
Alexey Veretennikov -
Faré