Hey,
I've finished running cl-test-grid on the rc2 (I hope I've fixed the issue – it wasn't the RNG, but defsetf bug).
Could you Anton create reports based on these new tests? Thank you! There is no rush with it of course.
Best regards, Daniel
12.02.2016, 20:58, "Daniel Kochmański" daniel@turtleware.eu:
Hey,
I've finished running cl-test-grid on the rc2 (I hope I've fixed the issue – it wasn't the RNG, but defsetf bug).
Could you Anton create reports based on these new tests? Thank you! There is no rush with it of course.
Here they are: https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc2-lin-x64... https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc2-lin-x64...
They are generated by these two calls: https://github.com/cl-test-grid/cl-test-grid-work/blob/1211e0f191542f3bb1cde...
Also, whey I sent you reports for RC1, I made a mistake, they were both generated from bytecode compiler. Sorry about that:
08.02.2016, 13:22, "Anton Vodonosov" avodonosov@yandex.ru:
Here are the diff reports:
https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-lin-x64-lis... https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-lin-x64-byt...
The fixed lisp-to-c diff for the previous RC: https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-lin-x64-lis...
Best regards, - Anton
Hey,
thank you for response.
Anton Vodonosov writes:
12.02.2016, 20:58, "Daniel Kochmański" daniel@turtleware.eu:
Hey,
I've finished running cl-test-grid on the rc2 (I hope I've fixed the issue – it wasn't the RNG, but defsetf bug).
Could you Anton create reports based on these new tests? Thank you! There is no rush with it of course.
Here they are: https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc2-lin-x64... https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc2-lin-x64...
Thanks. Weird, I've explicitly checked alexandria and there seems to be no failed tests on my machine. Maybe I had some cache leftovers? Or some other weird issue (like providing wrong path for the implementation). Not sure. Tested it right now with (asdf:test-system 'alexandria) to confirm that and I have "No tests failed". I'll rerun tests on Monday.
They are generated by these two calls: https://github.com/cl-test-grid/cl-test-grid-work/blob/1211e0f191542f3bb1cde...
Sorry for bothering you. I'll generate next reports myself :-)
Also, whey I sent you reports for RC1, I made a mistake, they were both generated from bytecode compiler. Sorry about that:
08.02.2016, 13:22, "Anton Vodonosov" avodonosov@yandex.ru:
Here are the diff reports:
https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-lin-x64-lis... https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-lin-x64-byt...
The fixed lisp-to-c diff for the previous RC: https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-lin-x64-lis...
Np. Thanks once again.
Best regards,
- Anton
Regards, Daniel
14.02.2016, 02:07, "Daniel Kochmański" daniel@turtleware.eu:
They are generated by these two calls: https://github.com/cl-test-grid/cl-test-grid-work/blob/1211e0f191542f3bb1cde...
Sorry for bothering you. I'll generate next reports myself :-)
No problem for me - let me know what are you doing next and how can I help. I just shared this script for better visibility, how cl-test-grid works.
Best regards, - Anton
OK, I've found the problem. I've added -rc2 suffix while forgetting to merge with the develop branch (silly me), while manually checking with my build from develop branch.
Regards, Daniel
Anton Vodonosov writes:
14.02.2016, 02:07, "Daniel Kochmański" daniel@turtleware.eu:
They are generated by these two calls: https://github.com/cl-test-grid/cl-test-grid-work/blob/1211e0f191542f3bb1cde...
Sorry for bothering you. I'll generate next reports myself :-)
No problem for me - let me know what are you doing next and how can I help. I just shared this script for better visibility, how cl-test-grid works.
Best regards,
- Anton
*sigh*, I've run the tests only against -rc3 on the server and it lacks some libraries and is too slow for some libraries to load (timeouts). I'm re-running both 16.0.0 and 16.1.2-rc3 on my noisy desktop machine :)
Regards, Daniel
PS I've succesfully generated the reports, though I had to find the version string by analizing result of
; -- (prog1 nil (setf *xxx* (remove-if-not #'(lambda (result) (string-equal "ecl" (lisp result) :end2 3)) *all-results*))) (remove-duplicates *xxx* :key #'lisp :test #'equalp) ; --
Daniel Kochmański writes:
OK, I've found the problem. I've added -rc2 suffix while forgetting to merge with the develop branch (silly me), while manually checking with my build from develop branch.
Regards, Daniel
Anton Vodonosov writes:
14.02.2016, 02:07, "Daniel Kochmański" daniel@turtleware.eu:
They are generated by these two calls: https://github.com/cl-test-grid/cl-test-grid-work/blob/1211e0f191542f3bb1cde...
Sorry for bothering you. I'll generate next reports myself :-)
No problem for me - let me know what are you doing next and how can I help. I just shared this script for better visibility, how cl-test-grid works.
Best regards,
- Anton
15.02.2016, 19:23, "Daniel Kochmański" daniel@turtleware.eu:
*sigh*, I've run the tests only against -rc3 on the server and it lacks some libraries and is too slow for some libraries to load (timeouts). I'm re-running both 16.0.0 and 16.1.2-rc3 on my noisy desktop machine :)
Regards, Daniel
PS I've succesfully generated the reports, though I had to find the version string by analizing result of
; -- (prog1 nil (setf *xxx* (remove-if-not #'(lambda (result) (string-equal "ecl" (lisp result) :end2 3)) *all-results*))) (remove-duplicates *xxx* :key #'lisp :test #'equalp) ; --
I usually look them up here: https://groups.google.com/forum/#!forum/cl-test-grid-notifications
Is there a way to pick the results if the commit and version name were the same? I've re-run the tests on my localhost machine, but the generated report uses results from the previous run. If not, I'll re-run the tests with the dummy -rc4 tag..
Regards, Daniel
Anton Vodonosov writes:
15.02.2016, 19:23, "Daniel Kochmański" daniel@turtleware.eu:
*sigh*, I've run the tests only against -rc3 on the server and it lacks some libraries and is too slow for some libraries to load (timeouts). I'm re-running both 16.0.0 and 16.1.2-rc3 on my noisy desktop machine :)
Regards, Daniel
PS I've succesfully generated the reports, though I had to find the version string by analizing result of
; -- (prog1 nil (setf *xxx* (remove-if-not #'(lambda (result) (string-equal "ecl" (lisp result) :end2 3)) *all-results*))) (remove-duplicates *xxx* :key #'lisp :test #'equalp) ; --
I usually look them up here: https://groups.google.com/forum/#!forum/cl-test-grid-notifications
16.02.2016, 18:30, "Daniel Kochmański" daniel@turtleware.eu:
Is there a way to pick the results if the commit and version name were the same? I've re-run the tests on my localhost machine, but the generated report uses results from the previous run. If not, I'll re-run the tests with the dummy -rc4 tag..
I've just filtered out old results by their test-run-time: https://github.com/cl-test-grid/cl-test-grid-work/blob/e44d78215d6e90297cd9c...
I took the value of timestamp in the notification email: https://groups.google.com/d/msg/cl-test-grid-notifications/qlDB70TaSQo/L7sDG...
The resulting report: https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc3-lin-x64...
BTW, new results were only submitted for lisp-to-c compiler.
I did one change in the report: new ECL version - RC3 - is on the right hand side. This is the usual way, but in the previous report I mistakenly did it other way around.
Let me know how can I help.
Best regards, - Anton
OK, thanks. I think I have something wrong with my setup because I can't load the crashed packages even from SBCL or CCL (problems with hu.dwim.*). I'll create a fresh user account, download testgrid and run it (with a new rc tag), I hope it will work OK...
Regards, Daniel
Anton Vodonosov writes:
16.02.2016, 18:30, "Daniel Kochmański" daniel@turtleware.eu:
Is there a way to pick the results if the commit and version name were the same? I've re-run the tests on my localhost machine, but the generated report uses results from the previous run. If not, I'll re-run the tests with the dummy -rc4 tag..
I've just filtered out old results by their test-run-time: https://github.com/cl-test-grid/cl-test-grid-work/blob/e44d78215d6e90297cd9c...
I took the value of timestamp in the notification email: https://groups.google.com/d/msg/cl-test-grid-notifications/qlDB70TaSQo/L7sDG...
The resulting report: https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc3-lin-x64...
BTW, new results were only submitted for lisp-to-c compiler.
I did one change in the report: new ECL version - RC3 - is on the right hand side. This is the usual way, but in the previous report I mistakenly did it other way around.
Let me know how can I help.
Best regards,
- Anton
16.02.2016, 23:09, "Daniel Kochmański" daniel@turtleware.eu:
OK, thanks. I think I have something wrong with my setup because I can't load the crashed packages even from SBCL or CCL (problems with hu.dwim.*).
How do you test that, via cl-test-grid or in REPL?
I'll create a fresh user account, download testgrid and run it (with a new rc tag), I hope it will work OK...
OK. BTW, when configuring RC results, you could specify "ecl" storage for them (instead of the default "main").
Like here: https://github.com/cl-test-grid/cl-test-grid/blob/master/run-agent.sample.li...
Let me know in case of problems.
Best regards, - Anton
I've finnaly managed to run the test grid and compare the results \o/ :-)
I've investigated the differences (potential regressions) by hand, and it seems that they behave the same on both ECL versions (short summary):
| | 16.0.0-c | 16.1.2-c | | cl-openid | operation not permitted (OS) | operation not permitted (OS) | | clack-errors | connection refused | connection refused | | quickutil | all fine | all fine | |--------------+------------------------------+------------------------------| | | 16.0.0-b | 16.1.2-b | | drakma | segment-violation | segment-violation | | i-type | frame-stack-overflow | frame-stack-overflow |
Some differences are present regarding handling the conditions. 16.1.2 catches overflow better with a proper restarts instead of jumping straight into the toplevel, but the causing error is the same. Reports attached.
I still had some problems with the batch run of cl-test-grid and had some minor confusions (SBCL as a preffered lisp signalled a condition once during the run, something with FIND-SYMBOL – sorry, I didn't preserve the log unfortunately. I have SBCL 1.3.1 as a preferred lisp in #'make-agent.
Also, after running for four implementations only three results were submitted (the last one was cached though, so I hadn't to re-run the tests, it submitted it right away).
Best regards, Daniel
Anton Vodonosov writes:
16.02.2016, 23:09, "Daniel Kochmański" daniel@turtleware.eu:
OK, thanks. I think I have something wrong with my setup because I can't load the crashed packages even from SBCL or CCL (problems with hu.dwim.*).
How do you test that, via cl-test-grid or in REPL?
I'll create a fresh user account, download testgrid and run it (with a new rc tag), I hope it will work OK...
OK. BTW, when configuring RC results, you could specify "ecl" storage for them (instead of the default "main").
Like here: https://github.com/cl-test-grid/cl-test-grid/blob/master/run-agent.sample.li...
Let me know in case of problems.
Best regards,
- Anton
Hi Daniel,
Thanks for trying all that. Great to hear no regressions detected, congrats.
I published the reports online at https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc4-lin-x64... https://common-lisp.net/project/cl-test-grid/ecl/ecl-diff-16.1.2-rc4-lin-x64...
18.02.2016, 11:40, "Daniel Kochmański" daniel@turtleware.eu:
I've investigated the differences (potential regressions) by hand, and it seems that they behave the same on both ECL versions (short summary):
...
| | 16.0.0-c | 16.1.2-c | | quickutil | all fine | all fine |
So, quickutil loads OK manualy, despite if failed under cl-test-grid? Most likely it's not caused by cl-test-grid. It may depend on presence and order of fasl files load; or even some undeterministic behavior in ECL. (It's just FYI, I understand it's not a regression since we don't see a difference between two ECL versions).
I still had some problems with the batch run of cl-test-grid and had some minor confusions (SBCL as a preffered lisp signalled a condition once during the run, something with FIND-SYMBOL – sorry, I didn't preserve the log unfortunately. I have SBCL 1.3.1 as a preferred lisp in #'make-agent.
Strange, I often use SBCL as a preferred lisp (not 1.3.1, elder versions). I will keep an eye on that.
Also, after running for four implementations only three results were submitted (the last one was cached though, so I hadn't to re-run the tests, it submitted it right away).
Rarely submit can fail (e.g. network problem - agent log could show the reason), and yes, the tests continue after restart from the point of interruption.
(Actually, there is even more rare bug - there is one point where during submit process where interruption prevents re-submit after restart; But so far I've been submitting manually in this special case).
Thanks for all the notes.
Best regards, - Anton
-
Anton Vodonosov -
Daniel Kochmański