I've updated the [CHANGES][1] file on trunk in preparation for the upcoming release. Erik asked me to write up my changes, but I took at stab at recapitulating everything as best I could interpret it.
[1]: http://code.google.com/p/abcl-dynamic-install/source/detail?r=8a0bf88cca6dde...
Deliberately missing from this summary:
1) Alessio's recent work on lambda inlining. In my opinion we have to address the bug Alan Ruttenberg has found or backout these changes before release.
2) Ville's work on string functions, as it's incomplete. And it was hard to understand from the commit messages if it is purely optimization or if it involves bug fixing as well.
Anything else is something I missed.
I've adopted a little more verbose format than previously, including references to svn changesets and tickets where possible. There are now three sections: "Features" which tries to summarize the features that would be interesting to end users "Fixes/Optimizations" summarizing all the fixes or optimizations that might prompt an end user to retry something that previously didn't work or was too slow, and "Other" which mostly summarizes changes to implementation details of interest to ABCL development.
Comments (or direct changes in SVN) are welcome.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 22 February 2010 12:25, Mark Evenson evenson@panix.com wrote:
Deliberately missing from this summary: 2) Ville's work on string functions, as it's incomplete. And it was hard to understand from the commit messages if it is purely optimization or if it involves bug fixing as well.
That work is complete now. It was just a cleanup of the code, in order to make StringFunctions.java less repetitive and hopefully more readable. It's perhaps not something that users should notice, In that sense it's not a changelog highlight.
On Mon, Feb 22, 2010 at 11:25 AM, Mark Evenson evenson@panix.com wrote:
- Alessio's recent work on lambda inlining. In my opinion we have to address the bug Alan Ruttenberg has found or backout these changes before release.
I've been silent on the issue, but that doesn't mean I'm not working on it :) I still haven't been able to find out what's wrong, but there's a potential clue. My changes break one ANSI test, apparently because I do a copy-tree I shouldn't do. (without the copy-tree the compiler crashed, so I need to pinpoint exactly what and when to copy). That may be unrelated with Alan's bug, but nevertheless I'll try to fix it. I have the feeling, not backed out by actual evidence, that Alan's bug is related only to the runtime compiler, which somewhat suggests that that copy-tree might have something to do with it as well. So, stay tuned. And in case I don't solve it in time for the release, no problem - it's easy to disable the optimization without reverting anything.
Bye, Alessio
On Mon, Feb 22, 2010 at 5:09 PM, Alessio Stalla alessiostalla@gmail.com wrote:
On Mon, Feb 22, 2010 at 11:25 AM, Mark Evenson evenson@panix.com wrote:
- Alessio's recent work on lambda inlining. In my opinion we have to address the bug Alan Ruttenberg has found or backout these changes before release.
I've been silent on the issue, but that doesn't mean I'm not working on it :) I still haven't been able to find out what's wrong, but there's a potential clue. My changes break one ANSI test, apparently because I do a copy-tree I shouldn't do. (without the copy-tree the compiler crashed, so I need to pinpoint exactly what and when to copy). That may be unrelated with Alan's bug, but nevertheless I'll try to fix it. I have the feeling, not backed out by actual evidence, that Alan's bug is related only to the runtime compiler, which somewhat suggests that that copy-tree might have something to do with it as well. So, stay tuned. And in case I don't solve it in time for the release, no problem - it's easy to disable the optimization without reverting anything.
I think I've fixed it. It was indeed a copy-tree, but not the one I thought (incidentally that one was only in my local copy, probably left over from a previous experiment). In one case the expanded function body shared structure with the form that generated the lambda call, and with compiler macros, like in Alan's case, that's really bad, as the code of the compiler macro gets destructively modified by the compilation process (at least, in my understanding), so I added a copy-tree for the function body. I'm not completely sure that copy-tree isn't copying too much, but it was already used in other places for the same thing before my changes. It seems now that read-obo works like it should, and the ANSI tests still pass, so I committed the change. Still, since it was quite a challenge for me to fix it, and it left me almost in a catatonic state :D, I may have missed something: please check that your code still works after the commit.
Alessio
On Tue, Feb 23, 2010 at 6:47 PM, Alessio Stalla alessiostalla@gmail.com wrote:
On Mon, Feb 22, 2010 at 5:09 PM, Alessio Stalla alessiostalla@gmail.com wrote:
On Mon, Feb 22, 2010 at 11:25 AM, Mark Evenson evenson@panix.com wrote:
- Alessio's recent work on lambda inlining. In my opinion we have to address the bug Alan Ruttenberg has found or backout these changes before release.
I've been silent on the issue, but that doesn't mean I'm not working on it :) I still haven't been able to find out what's wrong, but there's a potential clue. My changes break one ANSI test, apparently because I do a copy-tree I shouldn't do. (without the copy-tree the compiler crashed, so I need to pinpoint exactly what and when to copy). That may be unrelated with Alan's bug, but nevertheless I'll try to fix it. I have the feeling, not backed out by actual evidence, that Alan's bug is related only to the runtime compiler, which somewhat suggests that that copy-tree might have something to do with it as well. So, stay tuned. And in case I don't solve it in time for the release, no problem - it's easy to disable the optimization without reverting anything.
I think I've fixed it. It was indeed a copy-tree, but not the one I thought (incidentally that one was only in my local copy, probably left over from a previous experiment). In one case the expanded function body shared structure with the form that generated the lambda call, and with compiler macros, like in Alan's case, that's really bad, as the code of the compiler macro gets destructively modified by the compilation process (at least, in my understanding), so I added a copy-tree for the function body. I'm not completely sure that copy-tree isn't copying too much, but it was already used in other places for the same thing before my changes. It seems now that read-obo works like it should, and the ANSI tests still pass, so I committed the change. Still, since it was quite a challenge for me to fix it, and it left me almost in a catatonic state :D, I may have missed something: please check that your code still works after the commit.
Just tested my code. Looks good. Thanks for the brain cells! -Alan
On Feb 24, 2010, at 12:47 AM, Alessio Stalla wrote:
On Mon, Feb 22, 2010 at 5:09 PM, Alessio Stalla alessiostalla@gmail.com wrote:
On Mon, Feb 22, 2010 at 11:25 AM, Mark Evenson evenson@panix.com wrote:
- Alessio's recent work on lambda inlining. In my opinion we have to address the bug Alan Ruttenberg has found or backout these changes before release.
[…]
I think I've fixed it.
Congratulations! Hopefully you gained some understanding of the compiler out of this, that can be successfully applied in the future. I have yet to be brave enough to undertook non-trivial modifications of that code to go through a similar trial by fire.
[…]
It seems now that read-obo works like it should, and the ANSI tests still pass, so I committed the change. Still, since it was quite a challenge for me to fix it, and it left me almost in a catatonic state :D, I may have missed something: please check that your code still works after the commit.
Everything seems to work, but I notice that I seem to have 35 failing ANSI compiled tests, whereas I though we only should have 34.
N.B. We should really mark the expected failures because chasing the set differences of these failure lists always gives me headaches for what should be a simple task.
(REINITIALIZE-INSTANCE.ERROR.1 DEFGENERIC.ERROR.20 DEFGENERIC.ERROR.21 DEFGENERIC.30 CALL-NEXT-METHOD.ERROR.1 CALL-NEXT-METHOD.ERROR.2 DEFMETHOD.ERROR.14 DEFMETHOD.ERROR.15 INVOKE-DEBUGGER.1 MAKE-CONDITION.3 MAKE-CONDITION.4 DELETE-PACKAGE.5 DELETE-PACKAGE.6 MAP.48 TYPE-OF.1 TYPE-OF.4 CHAR-UPCASE.2 CHAR-DOWNCASE.2 ENSURE-DIRECTORIES-EXIST.8 FRESH-LINE.5 PRINT.RANDOM-STATE.1 PPRINT-FILL.14 PPRINT-FILL.15 PPRINT-LINEAR.14 PPRINT-TABULAR.13 PPRINT-LOGICAL-BLOCK.17 PPRINT-POP.7 PPRINT-POP.8 FORMAT.LOGICAL-BLOCK.CIRCLE.1 FORMAT.LOGICAL-BLOCK.CIRCLE.2 FORMAT.LOGICAL-BLOCK.CIRCLE.3 FORMAT.JUSTIFY.30 FORMAT.JUSTIFY.32 WITH-STANDARD-IO-SYNTAX.23 TRACE.8)
This is with r12506 which includes a few more changes then just . It doesn't look like Alessio's commit in r12505 to fix the lambda inlining is the culprit as r12503 also has 35 failures. Regression testing now with "hg bisect"… but if 35 is the "correct" number please someone speak up.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 24 February 2010 17:55, Mark Evenson evenson@panix.com wrote:
Everything seems to work, but I notice that I seem to have 35 failing ANSI compiled tests, whereas I though we only should have 34. N.B. We should really mark the expected failures because chasing the set differences of these failure lists always gives me headaches for what should >be a simple task.
It's not so difficult to compare failure lists. Dump the ansi test results into files and diff.
(REINITIALIZE-INSTANCE.ERROR.1 DEFGENERIC.ERROR.20 DEFGENERIC.ERROR.21 DEFGENERIC.30 CALL-NEXT-METHOD.ERROR.1 CALL-NEXT-METHOD.ERROR.2 DEFMETHOD.ERROR.14 DEFMETHOD.ERROR.15 INVOKE-DEBUGGER.1 MAKE-CONDITION.3 MAKE-CONDITION.4 DELETE-PACKAGE.5 DELETE-PACKAGE.6 MAP.48 TYPE-OF.1 TYPE-OF.4 CHAR-UPCASE.2 CHAR-DOWNCASE.2 ENSURE-DIRECTORIES-EXIST.8 FRESH-LINE.5 PRINT.RANDOM-STATE.1 PPRINT-FILL.14 PPRINT-FILL.15 PPRINT-LINEAR.14 PPRINT-TABULAR.13 PPRINT-LOGICAL-BLOCK.17 PPRINT-POP.7 PPRINT-POP.8 FORMAT.LOGICAL-BLOCK.CIRCLE.1 FORMAT.LOGICAL-BLOCK.CIRCLE.2 FORMAT.LOGICAL-BLOCK.CIRCLE.3 FORMAT.JUSTIFY.30 FORMAT.JUSTIFY.32 WITH-STANDARD-IO-SYNTAX.23 TRACE.8)
INVOKE-DEBUGGER.1 and WITH-STANDARD-IO-SYNTAX.23 shouldn't be there - they didn't fail in my baseline test (taken on 0.18 release) and they didn't fail the last time I ran the tests on trunk.
On 2/24/10 5:25 PM, Ville Voutilainen wrote:
On 24 February 2010 17:55, Mark Evensonevenson@panix.com wrote:
Everything seems to work, but I notice that I seem to have 35 failing ANSI compiled tests, whereas I though we only should have 34. N.B. We should really mark the expected failures because chasing the set differences of these failure lists always gives me headaches for what should>be a simple task.
It's not so difficult to compare failure lists. Dump the ansi test results into files and diff.
Explicitly marking which tests are expected to fail has the property that one doesn't have to compare anything with anything. Checking the tests would then provide immediate, unmistakable feedback.
I wrote the attached utility based on SET-DIFFENCE to parse and report on a simple s-expr based database of test results, and I have also attached the tests results I have collected over the last day. These results now contradict my earlier email, showing:
CL-USER> (difference 'compileit 'r12506 '0.18.0) R12506: 34 failures NIL 0.18.0: 35 failures (MAKE-BROADCAST-STREAM.8) NIL CL-USER> (difference 'doit 'r12506 '0.18.0) R12506: 33 failures NIL 0.18.0: 34 failures (MAKE-BROADCAST-STREAM.8)
which shows we actually improved our coverage in this development cycle (Ville fixed MAKE-BROADCAST-STREAM.8 in r12397).
INVOKE-DEBUGGER.1 and WITH-STANDARD-IO-SYNTAX.23 shouldn't be there - they didn't fail in my baseline test (taken on 0.18 release) and they didn't fail the last time I ran the tests on trunk.
My testing shows that the WITH-STANDARD-IO-SYNTAX.23 has been present since 0.18.0. Perhaps this is an OSX specific thing? I'll test that out next.
On 25 February 2010 19:35, Mark Evenson evenson@panix.com wrote:
I wrote the attached utility based on SET-DIFFENCE to parse and report on a simple s-expr based database of test results, and I have also attached the tests results I have collected over the last day. These results now contradict my earlier email, showing: CL-USER> (difference 'compileit 'r12506 '0.18.0) R12506: 34 failures NIL 0.18.0: 35 failures (MAKE-BROADCAST-STREAM.8) NIL CL-USER> (difference 'doit 'r12506 '0.18.0) R12506: 33 failures NIL 0.18.0: 34 failures (MAKE-BROADCAST-STREAM.8)
Looks good. Store this utility somewhere in our source tree, please?
Is it known when the WITH-STANDARD-IO-SYNTAX.23 regression started?
(LET ((*PRINT-PPRINT-DISPATCH* (COPY-PPRINT-DISPATCH NIL))) (SET-PPRINT-DISPATCH (QUOTE SYMBOL) (FUNCTION (LAMBDA (STREAM OBJ) (DECLARE (IGNORE OBJ)) (WRITE-STRING "FOO" STREAM)))) (LIST (LET ((*PRINT-PRETTY* T)) (PRINC-TO-STRING (QUOTE BAR))) (WITH-STANDARD-IO-SYNTAX (LET ((*PRINT-PRETTY* T)) (PRINC-TO-STRING (QUOTE BAR)))))) ("FOO" "FOO")
It should say ("FOO" "BAR")
It might be when the Stream code was being done?
On Thu, Feb 25, 2010 at 10:36 AM, Ville Voutilainen ville.voutilainen@gmail.com wrote:
On 25 February 2010 19:35, Mark Evenson evenson@panix.com wrote:
I wrote the attached utility based on SET-DIFFENCE to parse and report on a simple s-expr based database of test results, and I have also attached the tests results I have collected over the last day. These results now contradict my earlier email, showing: CL-USER> (difference 'compileit 'r12506 '0.18.0) R12506: 34 failures NIL 0.18.0: 35 failures (MAKE-BROADCAST-STREAM.8) NIL CL-USER> (difference 'doit 'r12506 '0.18.0) R12506: 33 failures NIL 0.18.0: 34 failures (MAKE-BROADCAST-STREAM.8)
Looks good. Store this utility somewhere in our source tree, please?
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On 26 February 2010 20:20, dmiles@users.sourceforge.net logicmoo@gmail.com wrote:
Is it known when the WITH-STANDARD-IO-SYNTAX.23 regression started?
I thought (and said to Mark) that it shouldn't fail - I was obviously wrong, it's a new test that I didn't have. I updated my ansi-tests and it does indeed fail. We need to test it on abcl 0.17 (as an example) to verify that it's old, but according to our investigations it looks like it's a matter of our with-standard-io-syntax failing to restore (or lexically bind) the pprint-dispatch table to the standard value.
Ville, yes correct.
this seems to fix it
diff -r 782582237d7c src/org/armedbear/lisp/with-standard-io-syntax.lisp --- a/src/org/armedbear/lisp/with-standard-io-syntax.lisp +++ b/src/org/armedbear/lisp/with-standard-io-syntax.lisp @@ -45,6 +45,7 @@ (*print-level* nil) (*print-lines* nil) (*print-miser-width* nil) + (*print-pprint-dispatch* (copy-pprint-dispatch nil)) (*print-pretty* nil) (*print-radix* nil) (*print-readably* t)
On Fri, Feb 26, 2010 at 11:51 AM, Ville Voutilainen ville.voutilainen@gmail.com wrote:
On 26 February 2010 20:20, dmiles@users.sourceforge.net logicmoo@gmail.com wrote:
Is it known when the WITH-STANDARD-IO-SYNTAX.23 regression started?
I thought (and said to Mark) that it shouldn't fail - I was obviously wrong, it's a new test that I didn't have. I updated my ansi-tests and it does indeed fail. We need to test it on abcl 0.17 (as an example) to verify that it's old, but according to our investigations it looks like it's a matter of our with-standard-io-syntax failing to restore (or lexically bind) the pprint-dispatch table to the standard value.
"dmiles@users.sourceforge.net" logicmoo@gmail.com writes:
Is it known when the WITH-STANDARD-IO-SYNTAX.23 regression started?
(LET ((*PRINT-PPRINT-DISPATCH* (COPY-PPRINT-DISPATCH NIL))) (SET-PPRINT-DISPATCH (QUOTE SYMBOL) (FUNCTION (LAMBDA (STREAM OBJ) (DECLARE (IGNORE OBJ)) (WRITE-STRING "FOO" STREAM)))) (LIST (LET ((*PRINT-PRETTY* T)) (PRINC-TO-STRING (QUOTE BAR))) (WITH-STANDARD-IO-SYNTAX (LET ((*PRINT-PRETTY* T)) (PRINC-TO-STRING (QUOTE BAR)))))) ("FOO" "FOO")
It should say ("FOO" "BAR")
It might be when the Stream code was being done?
I think WITH-STANDARD-IO-SYNTAX.23 may have been added recently due to a patch from mine. I discovered that SBCL does not bind *PPRINT-DISPATCH-TABLE* in W-ST-IO-SYNTAX a couple of months ago (and ABCL copied the pretty-printer stuff from SBCL), and I may have submitted a test case to the ansi test suite -- although I can't remember precisely. Try to annotate the ansi test source file.
-T.
On 28 February 2010 01:20, Tobias C. Rittweiler tcr@freebits.de wrote:
I think WITH-STANDARD-IO-SYNTAX.23 may have been added recently due to a patch from mine. I discovered that SBCL does not bind *PPRINT-DISPATCH-TABLE* in W-ST-IO-SYNTAX a couple of months ago (and ABCL copied the pretty-printer stuff from SBCL), and I may have submitted a test case to the ansi test suite -- although I can't remember precisely. Try to annotate the ansi test source file.
The problem has been fixed already, in r12510.
On 2/25/10 7:36 PM, Ville Voutilainen wrote: […]
Looks good. Store this utility somewhere in our source tree, please?
A substantially improved version that allows for multiple test results for a given source revision [has been committed][1]. As part of this commit, I put my database of failure in the tree with the intention that other developers can contribute their results, so we can have a more comprehensive picture.
I'm not entirely satisfied with the structure of the utility, particularly the output of REPORT, but now that the basics are in place, we can work on improving that.
[1]: http://trac.common-lisp.net/armedbear/changeset/12509
armedbear-devel@common-lisp.net
-
Alan Ruttenberg -
Alessio Stalla -
dmiles＠users.sourceforge.net -
Mark Evenson -
Tobias C. Rittweiler -
Ville Voutilainen