Vladmir, the library content is not just the signal trick (or special vars).
It defines the unified common test result representation, provides a bridge between the protocol test libraries "signal" their result and the protocol the user running tests consumes the results. The library can also provide means to create the common test results as a normal return value; or maybe deliver it through a stack-based mechanism, but explicitly, without relying on test frameworks to do it automatically (in some cases I anticipate explicit creation of the result will be more convenient)
And yes, the most important, adding such functionality into the public ASDF API requires too much upfront thinking. After separate library experiments with the approach and stabilizes, ASDF can incorporate, or officially refer it.
As Robert mentions in the pull request, having this functionality in new versions of ASDF would require the consumer to check feature flags in their code. In the case of a separate library, the consumer can just load this library, and it will work even with old versions of ASDF.
04.10.2019, 06:39, "Vladimir Sedach" vas@oneofus.la:
Robert Goldman rpgoldman@sift.info writes:
- success should also be signaled, so we can distinguish a version where this new protocol is not implemented from the version where tests pass
This requires a protocol where ASDF can "know" when the test op is done, so that it can distinguish "has succeeded" from "has not failed yet." It's possible that this could be managed by the test op on the system as a whole, but the design must consider the possibility that the test-op may be implemented so that:
1. Multiple calls are made into the test library (possibly even multiple test libraries are used) 2. TEST-OP may involve invoking test operation on systems depended on (e.g., where a system has subsystems).
There would be three situations here:
- OPERATE TEST-OP returns with no relevant conditions signaled. You
can infer that the condition protocol is not implemented. 2. OPERATE TEST-OP returns and one or more test failure conditions are signaled. 3. OPERATE TEST-OP returns and only test success conditions are signaled.
- the minimal requirement is a success / failure designator, the failed test names can be optional
- Additional requirement: the condition should support accessing both a long and a short form report. In degenerate implementations, these can, of course, be identical.
What would the long and short form reports look like?
some corner cases. Still, even if we expect test results being signalled multiple times during a test-op, it would be good to provide a wrapper which aggregates them into a single return value.
(common-test-results:collect (asdf:test-system "my-system"))
This would require that the test library provide an extensible protocol for fusing together multiple test results.
It is simpler than that: take all of the conditions, add the number of tests executed, and append all of the test failure lists. No need for library-specific code.
And note that the above suggestion will not work, because ASDF does not ever *return* a value from operating. This has to do with the way ASDF creates a plan and then executes it. The plan doesn't support a notion of "return value," so the only way to get information out of ASDF is through conditions.
What COMMON-TEST-RESULTS:COLLECT would do is handle and coalesce multiple conditions and re-signal a single condition.
I agree -- I think `TRIVIAL-TEST-INTERFACE` might be a better first step. I suppose the alternative rationale is that a test interface that was *not* intended for incorporation into ASDF would be able to just *return* things, instead of *signaling* them.
The point is, systems already define TEST-OP. I am trying to use that. The code that uses TEST-OP can do whatever it needs to, but the communication between that code and the test libraries has to be done by stack-based mechanisms like conditions or special variables.
That is true, but it's also true that it would require special condition-handling to fit test results into continuous integration -- programmers would no longer be able to just use `quit-on-error`, which is a very handy way to turn a lisp test into something that works in Jenkins or, for that matter, any infrastructure based on shell scripting.
Right now errors are not signaled on test failures in most definitions of TEST-OP I looked at, so this is not something that is currently going on. Neither is this something that would stop working for anyone that has the signal-error-on-failure flags set for their test library, or is throwing errors explicitly.
I'd rather have to write code to handle errors when I *don't* want them, than have test failure not be an error.
If I'm running interactively, it's not a bother to deal with this as an error condition -- I can easily get out of the debugger. But writing a lot of code to catch `TEST-FAILURE` conditions and translate them into exit with non-zero status would be a pain.
Test libraries already have flags whether to signal errors on test failures or not. Having the condition be a sub-class of error would not only be annoying in the REPL, it would break whatever test automation code uses these flags, and it would change the behavior of TEST-OP, most of whose callers do not expect it to signal errors on test failures right now. This is a lot of breakage of thousands of existing systems, just to avoid doing the following in a few test automation scripts:
(handler-case (asdf:test-system "some-system") (asdf:test-op-test-failures (condition) (princ condition uiop:*stderr*) (uiop:quit 1)))
- slot for the failing asdf system could probably be avoided, the list failed test names could be enough, if the names are "fully qualified" i.e. include package or system name.
I don't think we can make any assumptions about the above -- there's no rule about how a programmer can assign test names in a library like FiveAM to packages.
FiveAM test names are symbols, so they already get printed with their package name in the implementation I did for FiveAM, without any extra work.
I would note also that getting a new library into Quicklisp for this is going to be a lot easier than getting a new ASDF into Quicklisp: Xach has for years refused to update the ASDF version in Quicklisp, and I don't see any reason to believe this will change.
As I mentioned before, I would like to avoid creating a whole library out of something that is a work-around to OPERATE not returning results.
Unfortunately, it seems there are both social and technical problems with updating ASDF. In particular I do not see a good mechanism for advertising the availability of this condition protocol to test libraries (there does not seem to be an established way of advertising new ASDF functionality other than the :ASDF3.3 :ASDF3.2 etc. keywords in *FEATURES*).
As Anton pointed out, this necessitates the libraries signaling a condition for test success, which necessitates a function like COMMON-TEST-RESULTS:COLLECT. While writing an implementation of that function, I realized I would need to add continue restarts (the only way to handle a condition without affecting control flow).
Take all of this together, and it becomes apparent that avoiding ASDF to provide a more useful TEST-OP is, ironically, the way to go. It is simpler to drive communications down the stack by binding special variables to act as accumulators, than it is communicating up the stack with signals, handlers, and restarts. Putting this into a library means TEST-OP would still retain a use as a way to trigger test runs without knowing details about either the tests or the test library (the first half of this proposal), but it will unfortunately mean that ASDF will continue to have no say about what the effects of TEST-OP are (the second half of this proposal).
-- Vladimir Sedach Software engineering services in Los Angeles https://oneofus.la