[cffi-devel] Unit tests failures on different lisps
Hello. I am running tests of some most often downloaded CL libraries, including CFFI. So far CFFI was only tested on Linux. It has Linux, has different number (of expected and unexpected) failures on different Lisps. The results may be found here: http://common-lisp.net/project/cl-test-grid/pivot_ql-lib_lisp.html Clicking the ok/fail status refer to the test logs with the failures details. Best regards, - Anton
Some test results from me: SBCL on Mac OSX: no unexpected failures ACL 8.2 x86_64 on Mac OSX: crash in test. This is the test: #-cffi-features:no-long-long (progn (defcfun "my_llabs" :long-long (n :long-long)) (deftest defcfun.long-long (my-llabs -9223372036854775807) 9223372036854775807)) I believe that ACL has no long long --- why is this test executed? I see this in cffi-allegro.lisp: #-64bit (pushnew 'no-long-long *features*) (pushnew 'flat-namespace *features*) Since I am running a 64-bit allegro, I believe that the #-64bit should simply be removed. Also, probably we should fix the test so that it fails, instead of crashing in compilation... With no-long-long added to the feature set, I see: 10 unexpected failures: FUNCALL.LONG-LONG, FUNCALL.VARARGS.DOUBLE, DEFCFUN.VARARGS.FLOAT, DEFCFUN.VARARGS.DOUBLE, DEFCFUN.BFF.1, POINTERP.4, POINTERP.5, POINTER-EQ.NON-POINTERS.1, POINTER-EQ.NON-POINTERS.2, NULL-POINTER-P.NON-POINTER.2. CCL 1.7 on Mac OSX: no unexpected failures Hope that helps, r
On Wed, Dec 28, 2011 at 9:25 PM, Anton Vodonosov <avodonosov@yandex.ru> wrote:
I am running tests of some most often downloaded CL libraries, including CFFI.
So far CFFI was only tested on Linux. It has Linux, has different number (of expected and unexpected) failures on different Lisps.
Thanks, this is rather helpful since these days I don't convenient access to all the Lisps/platforms supported by CFFI. Those failures are mostly harmless, although the BFF ones are probably implementation-specific bugs. Is there a way to take advantage of your buildfarm such that I can test CFFI versions before doing a release or before even pushing to the main repository? Cheers, -- Luís Oliveira http://r42.eu/~luis/
29.12.2011, 04:29, "Luís Oliveira" <luismbo@gmail.com>:
Is there a way to take advantage of your buildfarm such that I can test CFFI versions before doing a release or before even pushing to the main repository?
Not yet, because I am in the beginning of the project and only starting to see the first results (it's not really a buildfarm, but a small system where anyone may run a simple command, like (load "agent.lisp"), and it runs the tests and submits the results to the central server; so that we will be able to accumulate the statictics; currently this command just runs tests on whatever is the current quicklisp distribution is installed on the system, and I am the only person who runs this command - I am testing how it works and polishing the corner cases). The ability for library authors to express a wish to run tests on the recent (not released) version is one of the important future goals. There is no decision yet on how it will be specified, we may discuss it. I will contact you when (hopefully) I ready for this. Best regards, - Anton
On Thu, Dec 29, 2011 at 3:23 AM, Anton Vodonosov <avodonosov@yandex.ru> wrote:
Not yet, because I am in the beginning of the project and only starting to see the first results (it's not really a buildfarm, but a small system where anyone may run a simple command, like (load "agent.lisp"), and it runs the tests and submits the results to the central server; so that we will be able to accumulate the statictics; currently this command just runs tests on whatever is the current quicklisp distribution is installed on the system, and I am the only person who runs this command - I am testing how it works and polishing the corner cases).
OK, cool stuff. Have you considered using something like Jenkins? -- Luís Oliveira http://r42.eu/~luis/
29.12.2011, 15:08, "Luís Oliveira" <luismbo@gmail.com>:
On Thu, Dec 29, 2011 at 3:23 AM, Anton Vodonosov <avodonosov@yandex.ru> wrote:
Not yet, because I am in the beginning of the project and only starting to see the first results (it's not really a buildfarm, but a small system where anyone may run a simple command, like (load "agent.lisp"), and it runs the tests and submits the results to the central server; so that we will be able to accumulate the statictics; currently this command just runs tests on whatever is the current quicklisp distribution is installed on the system, and I am the only person who runs this command - I am testing how it works and polishing the corner cases).
OK, cool stuff. Have you considered using something like Jenkins?
Jenkins is an interesting software (thanks for the reference BTW), but it (and other continuous integration servers) has not so much overlap with what I do. For Jenkins to run tests we need to provide a command "run tests". Out of box it will be unable to run CL tests. So it's the first thing I do - unify the interface of running test suites of different CL libraries. Also it is important to represent results so that we can see side by side results of a particular library on different CL implementations; or results of all the libraries on different quicklisp distributions and so on. Continuous integration servers will not provide such a representation. Another goal is that people can contribute test results for their platforms. This will save us from dependency and maintenance of some central build farm with all the possible combinations of OS and CL implementations. So, while continue integration servers main function is to run some command - "run tests" - on some schedule or by evens like source control commits. In my opinion scheduling is not the main task to solve. With cl-test-grid I go bottom-up by providing an easy way to run tests on any platform and share results. If we have several contributors who agree to run a simple command manually say once a month, it should be enough (for the beginning at least) to monitor the CL landscape and detect regressions. After this will work, probably someone will want to configure a cron job for this command and don't spend any attention on this at alll, but on some platforms there might be limitations which will require starting tests manually. Today it's a bit early to think about this. Best regards, - Anton
On Wed, 28 Dec 2011 19:23:44 -0800, Anton Vodonosov <avodonosov@yandex.ru> wrote:
29.12.2011, 04:29, "Luís Oliveira" <luismbo@gmail.com>:
Is there a way to take advantage of your buildfarm such that I can test CFFI versions before doing a release or before even pushing to the main repository?
Not yet, because I am in the beginning of the project and only starting to see the first results (it's not really a buildfarm, but a small system where anyone may run a simple command, like (load "agent.lisp"), and it runs the tests and submits the results to the central server; so that we will be able to accumulate the statictics; currently this command just runs tests on whatever is the current quicklisp distribution is installed on the system, and I am the only person who runs this command - I am testing how it works and polishing the corner cases).
The ability for library authors to express a wish to run tests on the recent (not released) version is one of the important future goals. There is no decision yet on how it will be specified, we may discuss it. I will contact you when (hopefully) I ready for this.
Best regards, - Anton
This is really good. I wonder if it mightn't be easier to set up a number of VM's under something like VirtualBox in a way which would allow automated testing with simpler intra-platform communications. Can you publish any of your test scripts? I'd like to perform the experiment. Best regards, Jeff
06.01.2012, 18:13, "Jeffrey Cunningham" <jeffrey@jkcunningham.com>:
This is really good. I wonder if it mightn't be easier to set up a number of VM's under something like VirtualBox in a way which would allow automated testing with simpler intra-platform communications. Can you publish any of your test scripts? I'd like to perform the experiment.
Yes, please check the description here: https://github.com/cl-test-grid/cl-test-grid I've also set up a google group for discussions, we can continue there to avoid off-toping cffi-devel: https://groups.google.com/group/cl-test-grid Best regards, - Anton
participants (4)
-
Anton Vodonosov -
Jeffrey Cunningham -
Luís Oliveira -
Robert Goldman