Re: [babel-devel] [cffi-devel] Unit tests failures on different lisps
On Wed, Dec 28, 2011 at 9:25 PM, Anton Vodonosov <avodonosov@yandex.ru> wrote:
I am running tests of some most often downloaded CL libraries, including CFFI.
So far CFFI was only tested on Linux. It has Linux, has different number (of expected and unexpected) failures on different Lisps.
Thanks, this is rather helpful since these days I don't convenient access to all the Lisps/platforms supported by CFFI. Those failures are mostly harmless, although the BFF ones are probably implementation-specific bugs. Is there a way to take advantage of your buildfarm such that I can test CFFI versions before doing a release or before even pushing to the main repository? Cheers, -- Luís Oliveira http://r42.eu/~luis/
29.12.2011, 04:29, "Luís Oliveira" <luismbo@gmail.com>:
Is there a way to take advantage of your buildfarm such that I can test CFFI versions before doing a release or before even pushing to the main repository?
Not yet, because I am in the beginning of the project and only starting to see the first results (it's not really a buildfarm, but a small system where anyone may run a simple command, like (load "agent.lisp"), and it runs the tests and submits the results to the central server; so that we will be able to accumulate the statictics; currently this command just runs tests on whatever is the current quicklisp distribution is installed on the system, and I am the only person who runs this command - I am testing how it works and polishing the corner cases). The ability for library authors to express a wish to run tests on the recent (not released) version is one of the important future goals. There is no decision yet on how it will be specified, we may discuss it. I will contact you when (hopefully) I ready for this. Best regards, - Anton
On Thu, Dec 29, 2011 at 3:23 AM, Anton Vodonosov <avodonosov@yandex.ru> wrote:
Not yet, because I am in the beginning of the project and only starting to see the first results (it's not really a buildfarm, but a small system where anyone may run a simple command, like (load "agent.lisp"), and it runs the tests and submits the results to the central server; so that we will be able to accumulate the statictics; currently this command just runs tests on whatever is the current quicklisp distribution is installed on the system, and I am the only person who runs this command - I am testing how it works and polishing the corner cases).
OK, cool stuff. Have you considered using something like Jenkins? -- Luís Oliveira http://r42.eu/~luis/
29.12.2011, 15:08, "Luís Oliveira" <luismbo@gmail.com>:
On Thu, Dec 29, 2011 at 3:23 AM, Anton Vodonosov <avodonosov@yandex.ru> wrote:
Not yet, because I am in the beginning of the project and only starting to see the first results (it's not really a buildfarm, but a small system where anyone may run a simple command, like (load "agent.lisp"), and it runs the tests and submits the results to the central server; so that we will be able to accumulate the statictics; currently this command just runs tests on whatever is the current quicklisp distribution is installed on the system, and I am the only person who runs this command - I am testing how it works and polishing the corner cases).
OK, cool stuff. Have you considered using something like Jenkins?
Jenkins is an interesting software (thanks for the reference BTW), but it (and other continuous integration servers) has not so much overlap with what I do. For Jenkins to run tests we need to provide a command "run tests". Out of box it will be unable to run CL tests. So it's the first thing I do - unify the interface of running test suites of different CL libraries. Also it is important to represent results so that we can see side by side results of a particular library on different CL implementations; or results of all the libraries on different quicklisp distributions and so on. Continuous integration servers will not provide such a representation. Another goal is that people can contribute test results for their platforms. This will save us from dependency and maintenance of some central build farm with all the possible combinations of OS and CL implementations. So, while continue integration servers main function is to run some command - "run tests" - on some schedule or by evens like source control commits. In my opinion scheduling is not the main task to solve. With cl-test-grid I go bottom-up by providing an easy way to run tests on any platform and share results. If we have several contributors who agree to run a simple command manually say once a month, it should be enough (for the beginning at least) to monitor the CL landscape and detect regressions. After this will work, probably someone will want to configure a cron job for this command and don't spend any attention on this at alll, but on some platforms there might be limitations which will require starting tests manually. Today it's a bit early to think about this. Best regards, - Anton
participants (2)
-
Anton Vodonosov -
Luís Oliveira