On 7 Jun 2018, at 1:03, Mark Evenson wrote:

On Jun 7, 2018, at 00:21, Robert Goldman <rpgoldman@sift.info> wrote:

When I replaced fiveam-tester-system with :fiveam-tester-system or "FIVEAM-TESTER-SYSTEM", the load went fine for me with the latest ASDF on Allegro, SBCL, and ABCL 1.5.0.

I must have been somehow introducing errors by being non-rigorous in my testing yesterday because when I woke up fresh today to revisit this problem, I was indeed able to get both the keyword package designator and the string package designator forms to work under sbcl-1.4.8, ccl-1.11.5, and lispworks-pro-7.1.0.0.

I worked over my code to the point that I am ready to start integrating with my test harness invocation code.

Instead of using ERROR for raising the condition, I am [going to use SIGNAL][1]. This will mean that as long as no handler for ASDF:TEST-SUITE-FAILURES is installed, the test suite can be invoked by a developer normally. But when the automated test runner invokes the tests with an [appropiate handler installed][2], I’ll be able to make the test invocation “go red."

[1]: https://github.com/emotiq/asdf-test-harness/blob/master/asdf-test-harness.lisp#L42

[2]: https://github.com/emotiq/asdf-test-harness/blob/master/eg/run-with-handler.lisp

This sounds like an improvement. I may import it into the main version of fiveam-asdf.

Really, no one should be running with an ASDF that's older than 3.2 these days. I thought that all the OSS lisps were distributed with at least 3.2 now. Why are you running with 3.1.x?

I’m attempting to support a rather diverse set of Lisp hackers using a fair mix of development environments so it is organizationally easier for me to use the built-in ASDF. asdf-3.1.1 is what ccl-1.11.5 ships with, so it is one of versions I would like to support.

My intention for ASDF-TEST-HARNESS will be to provide a single mechanism to support test invocation for variety of differing test frameworks. The necessary [ASDF definition][3] will be quite simple, merely naming the test framework, and the package to test:

(defsystem "asdf-test-harness-example"
:defsystem-depends-on (asdf-test-harness)
:class :harness-test
:test-framework :lisp-unit
:test-package :lisp-unit-example
:components ((:file "lisp-unit-example")))

[3]: https://github.com/emotiq/asdf-test-harness/blob/master/eg/asdf-test-harness-example.asd

That sounds interesting. Note for some bigger systems we have had test operations that work over multiple packages (hence the longer form for the specification in the fiveam-tester-system class.

Almost all test frameworks which I have reviewed support a method to introspect a package for running the tests contained therein. All one will have to do to support a new framework will be write a specialization on a generic method which given the package name, runs the tests, collects the results, and returns a boolean truth value as to whether that test invocation should be considered pass (“Green”) or fail (“Red”).

OK, but note that you need a means of handling cases where running one test-system involves testing multiple subsidiary systems. In which case the boolean return will have to be handled carefully, because presumably you want your overall test to have AND logical semantics. You'll have to signal component system test failures and then have a handler that will defer them until all the tests are finally run.

Initially, I plan to support LISP-UNIT, PROVE, and FIVEAM.

This sounds great. We could talk about whether to continue to use asdf/contribs/ for this. Maybe not, since you will presumably want to maintain this code in your own repo.

Now, if I can only get Quicklisp to work again with :DEFSYSTEM-DEPENDS-ON clauses, I will be a pig in slop heaven…

I'm not sure what this issue is, but it's complicated by the fact that QL operates with such an old version of ASDF. In effect this means that QL will operate with whatever is the current ASDF in your environment, because ASDF won't downgrade itself. This means that people are running with all kinds of different ASDF versions, making debugging difficult.

Thanks very much for the help.

No problem!

Best,
r

--
"A screaming comes across the sky. It has happened before but there is nothing
to compare to it now."