Robert Goldman rpgoldman@sift.info writes:
Don't a lot of these CI systems just rely on the exit status of a build step? I typically just wrap a trivial script around running the test-op from fiveam-asdf, and make lisp exit with a non-zero status when it encounters an error. That's been sufficient for me, but then I tend to just look at success or failure, and read the console output on failures. I don't do anything fancy like generate XML...
That is what all of the CI scripts I have looked at in various projects do (there may be others I have not seen; please post them if you know of any). FiveAM itself provides a typical example: https://github.com/sionescu/fiveam/blob/master/.travis.yml#L40
(uiop:quit (if (some (lambda (x) (typep x '5am::test-failure)) (5am:run :it.bese.fiveam)) 1 0))
With my proposal becomes:
(handler-case (asdf:test-system "any-system") (asdf:test-op-test-failure (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
And now the lowest common denominator case for CI systems is automatic.
To expand on my previous statement that this proposal will not have backward-compatibility problems for CI systems because signals can be sub-classed, here is what that could look like:
Someone writes a GitLab continuous integration library, call it cl-gitlab-ci, and wants to add support to test libraries. cl-gitlab-ci defines a condition:
(define-condition cl-gitlab-ci-test-failure () ((test-failure-info …) (test-skip-info …)) …)
Libraries like FiveAM then add this condition as a subclass and fill in the necessary slots:
(define-condition test-spec-failure (cl-gitlab-ci-test-failure asdf:test-op-test-failure) …)
This does not break backward compatibility.
Same thing for a Travis integration library.
When someone decides to write a CI meta-library that abstracts over cl-gitlab-ci, cl-travis-ci, etc. (which as a basis will have what Stelian is asking for), they can define a condition class, generic-ci-test-failure, then send patches to the CI libraries to handle that condition, and send patches to the test libraries to subclass generic-ci-test-failure and fill out the slots (this is just like adding support for another CI library to the test library). Again, this does not break backward compatibility.
Currently popular CI programs include Buildbot, GitLab, Jenkins, Travis CI (essentially limited to GitHub). If someone has a proposal for a protocol that will cover the needed features for all of these systems, please post it.
Right now the only library for any of these programs that I know of is cl-travis. The hypothetical CI meta-library will ideally be built from experience with several (as of now non-existing) libraries for the various CI programs. Not only do I think that this is outside the scope of ASDF, I also suspect that designing a protocol for CI integration now, without the needed experience from actual libraries for various CI systems, is what is going to result in backward compatibility problems.
-- Vladimir Sedach Software engineering services in Los Angeles https://oneofus.la