Hello, thanks for the feedback
13.03.2012, 03:55, "Luís Oliveira" luismbo@gmail.com:
However, the results are a bit depressing. So many fails. :-)
Not so many - 20 failed tests in total.The same failures repeat on different lisps.
Here is the breakdown by failures, for the quicklisp 2012-02-08 and the lisp implementations we tested:
"callbacks.bff.1" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64" "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64" "cmu-20c_release-20c__20c_unicode_-linux-x86" "ecl-11.1.1-606449eb-linux-x86") "callbacks.bff.2" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64" "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64" "cmu-20c_release-20c__20c_unicode_-linux-x86") "callbacks.uninterned" => ("ecl-11.1.1-606449eb-linux-x86") "defcfun.bff.2" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-win-x86" "clisp-2.49-unix" "clisp-2.49-win" "cmu-20c_release-20c__20c_unicode_-linux-x86" "sbcl-1.0.54-linux-x86") "defcfun.noop" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64" "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64" "ecl-11.1.1-606449eb-linux-x86") "defcfun.stdcall.1" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win") "defcfun.undefined" => ("cmu-20c_release-20c__20c_unicode_-linux-x86") "defcfun.varargs.double" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win") "defcfun.varargs.float" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win") "foreign-symbol-pointer.1" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win") "funcall.stdcall.1" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win") "funcall.varargs.double" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win") "string.encoding.utf-16.basic" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64" "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64" "clisp-2.49-unix" "clisp-2.49-win" "cmu-20c_release-20c__20c_unicode_-linux-x86" "ecl-11.1.1-606449eb-linux-x86" "sbcl-1.0.49-linux-amd64" "sbcl-1.0.54-linux-x86" "sbcl-1.0.54.45-a2bef14-macosx-x64") "string.encodings.all.basic" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64" "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64" "clisp-2.49-unix" "clisp-2.49-win" "cmu-20c_release-20c__20c_unicode_-linux-x86" "ecl-11.1.1-606449eb-linux-x86" "sbcl-1.0.49-linux-amd64" "sbcl-1.0.54-linux-x86" "sbcl-1.0.54.45-a2bef14-macosx-x64") "struct.alignment.3" => ("ccl-1.7-f95-linux-x86") "struct.alignment.4" => ("ccl-1.7-f95-linux-x86") "struct.alignment.5" => ("ccl-1.7-f95-linux-x86") "struct.alignment.6" => ("ccl-1.7-f95-linux-x86") "struct.alignment.7" => ("ccl-1.7-f95-linux-x86") "struct.alignment.8" => ("ccl-1.7-f95-linux-x86")
Perhaps you could show the ration of failed to total tests
Can not do it now - we do not collect total number of tests in test suite. In my opinion it is not very useful information (comparing to the list of failed tests, which I hope will allow us to prevent new failures in future)
and maybe show know-fail/unexpected-ok in yellow/orange rather than red.
this is doable - added a TODO item
I wonder if compilation errors could be printed. Error messages like this are not very helpful: http://cl-test-grid.appspot.com/blob?key=AMIfv97suboJpeei-uBWzlkqcR7CTlyh0Izhvi7u_29HNBgu80ScYf0Mj6zWPjgbsosA-F0Q12HP8o9S5zhsEelTfss8_3C7sjgcuG_q_grR-jMfXPLLRzu6CNytLoNk23rwqlQ6AsajxTRYFubFbz3iBWl5uo8iZQ.
Lisp compilation failures are present in logs, but in this case we have C library compilation error. This may be seen by component description in the ASDF error message: #<C-TEST-LIB "cffi-tests" "tests" "libtest">
Actually, I try to represent this case - absence of the native library and therefore inability to run tests - by status :no-resource. But this failure is signaled differently on windows and linux, I didn't noticed this and in result in this linux test run was unable to recognize the failure as :no-resource.
Some details.
In cffi-tests.asd the
(defmethod perform ((o compile-op) (c c-test-lib))
does not try to run "make" on windows,
#-windows (unless (zerop (run-shell-command "cd ~A; make"
and on windows the only error signaled in absense of the native library is cffi:load-foreign-library-error. I rely on it when detecting :no-resource.
Added TODO item to detect is as :no-resouce on linux too when compile-op for the library fails.
Best regards, - Anton