June 2005 - iterate-devel - mailman3.common-lisp.net

[iterate-devel] final version of "conservative" Iterate: 1.4
by Hoehle, Joerg-Cyril 23 Jun '05

23 Jun '05

Hi, I've sent a new version of Iterate to Marco, Marc, Kevin, Peter and a few others, so it should appear in the usual places soon. Changes are too numerous to mention (and it depends on which version you're currently using:) and still extremely conservative. I'll have to supply the updated iter-man.pdf a little later, because my current Ubuntu laptop generates much uglier fonts therein than the old Suse one (maybe the default printer which is ljfour is the reason??). o supports nested in-hashtable and in-package o adjoin :at :start did not honour :test o fixed in-file/in-stream destructuring o do-next forgot about finally etc. clauses o works as case-sensitive package in CLISP (don't know about Allegro) o various optimizations, less bugs, more tests o tested with clisp, cmucl, sbcl, gclcvs This is what I consider the last version of 1.4, called "conservative". 1.5 will do some possibly incompatible changes to allow more optimizations (see recent e-mail of mine about my version plan). Regards, Jörg Höhle.

1 0

Re: [iterate-devel] Version plan for Iterate 1.4 and 1.5
by Hoehle, Joerg-Cyril 17 Jun '05

17 Jun '05

Marc Battyani wrote: >>code-movement [possible changes] >Those parts (prologue/epilog/var bindings) are under- (and >badly) specified >in LOOP and implementations differ on this. So let's keep it >in a sane way. I find it underspecified in Iterate as well. I want to change that state :-) E.g. Andreas' with-*-iterator based in-hashtable and in-package has very different initialization behaviour from the old ones (see below). It occured to me that there's another issue: iterate-pg may not work as expected, for similar reasons. (iter (with foo = connection) (with x on-relation foo connection bar)) #) ;won't work Reason: WITH-PG-TRANSACTION is move outside the whole Iterate form, thus foo and bar get evaluated before any other thing in Iterate(!) My current belief is that the following order would be a better match: (let (...) (wrappers: unwind-protect and with-pg-transaction, in order of apperance (block (tagbody #)))) Currently, it's (wrappers: with-package-iterator, with-hashtable-iterator, with-pg-transaction (let # (unwind-protect (block (tagbody #))))) Therefore (iter (with table = (foo)) (for (k v) in-hashtable table)) used to work with the old maphash based Iterate, but fails since Andreas' changes in 2004. I think this is also a result of underspecification of Iterate, together with a quite complex package and lots of feature interactions (initialization time, generators, declarations, top-level only clauses, support for PREVIOUS etc.) As a result, simple examples (e.g. the testsuite) work, but subtle forms fail. I'm going to change that as well (1.5). At the present, one can think of two forms of body wrappers: a) around the whole body (e.g. current with-*-iterator) b) around BLOCK, but inside LET (e.g. unwind-protect) I think I'm going to provide for only variant b, as shown above. Subtle interactions between unwind-protect clauses can still happen. I think that's a sign of limitations of Iterates design. All in all, packages needing a code-walker are not that great (they appear like a can of worms, e.g. Screamer was well, with some forms that work as expected and others that fail surprisingly (cf. primordial in Screamer). There's quite some truth when Bruno Haible expresses that CL somehow fails to be its own extension language. LOOP's weakness is also its strength: not needing a code walker, being limited to top-level forms, having a grammar so its transformation does not require macroexpand and remains syntactic. Regards, Jörg Höhle

1 0

[iterate-devel] Iterate in-hashtable expansion, with-hashtable-iterator vs. mapha sh
by Hoehle, Joerg-Cyril 15 Jun '05

15 Jun '05

Hi, first, some good news: I think I found an easy way to make Iterate support nested in-hashtable, together with with-hashtable-iterator: Just map with-hashtable-iterator to walk-cddr-with-declarations in *special-form-alist* That should basically work good enough for a start, but it does not walk the hashtable parameter. Next: a possible change -- what do you think? Some packages like SCREAMER have explicitly chosen to support Iterate instead of loop because they walk code and suffer from similar limitations about macrolet as Iterate does. In the past, Iterate never expanded to forms that Screamer was unable to analyse. Starting with Andreas Fuchs' change, he made IN-HASHTABLE expand to with-package-iterator, which is based on MACROLET. Similarly for in-package. As a result, Screamer would error out should it be updated to use one of ours iterate.lisp. Therefore, I'm thinking about introducing a choice, either as a *settable-variable*, or as new iteration keywords, for how to Iterate through hashtables (and packages). I first thought about in-hashtable: use with-hashtable-iterator on-hashtable: use maphash to build up list upfront. Then I realized that it should rather be the other way round, or somebody would have to modify the Screamer examples, and all other unknown packages built using Screamer (and possibly Series, I don't know). An alternative may be to get Screamer support macrolet, but that's IMHO even farther away than having Iterate do that. So what do you prefer? a) in-hashtable -> maphash, on-hashtable -> with-*-iterator and same for packages "reintroduce old behaviour" b) on-hashtable -> maphash, in-hashtable -> with-*-iterator and same for packages "affirm changed behaviour, need to change random packages" c) defvar iterate::*use-with-iterators* Screamer could set such a variable, or at least the examples should do so (since only the examples depend on Iterate, not Screamer itself) I'm biased towards b). Most users will benefit from the iterator, and I'd need to convince the Screamer authors anyway to package a more recent version of Iterate with it, so the examples could be updated at that time, or to at least have the debian cl-screamer package not include but depend on cl-iterate (and similarly for cl-pdf etc.). Still, the users would have to change their Screamer code. (and no, I haven't yet tested Screamer with my version of Iterate, this is just code review and thinking loud, yet I expect no problems beyond that). BTW, having a maphash-based iterator allows to modify any hashtable item while iterating it, instead of just the current one (forbidden by ANSI-CL and many other language specifications). That can come handy sometime. While we are at it, there's a difference between the old in-package and the new (since Andreas Fuchs in 2004): the new (with-package-iterator based) one does not evaluate the :external-only argument any more -- How I hate it when documentations does not precisely state what is evaluated and what not! it always makes simple examples work and complex ones fail. With-package-iterator does not provide for evaluation of this, while the old in-hashtable dispatched to either do-symbols or do-external-symbols. So far, I plan to stick to that change, or does anybody wave hands? Regards, Jörg Höhle.

1 0

Re: [iterate-devel] COMPILE FAILURE UNDER ALLEGRO CL 7
by Hoehle, Joerg-Cyril 13 Jun '05

13 Jun '05

Mark L. Williams CIV NSWC-PC wrote: >>;;; Compiling file [path]/iterate.lisp >>; While compiling (:top-level-form "iterate.lisp" 85868): >>Error: DEFCLAUSE: required part of template (FOR var-spec &sequence) >>is of odd length Since I am suspecting a "modern" Allegro package problem, could you please take a look at (defun define-clause (define-form clause-template body generator?) in the source? In my current source (since RCS 1.24, mid-April), it says therein: (if (and (symbolp last) (string= last '&sequence)) In iterate-1.0.9 and still in November 2004, it said: (if (and (symbolp last) (string= last "&SEQUENCE")) a) what does yours say? b) if "&SEQUENCE", does '&sequence help? BTW, I'm about to send Marc, Marco and Peter a new version. It would be nice to hear if b) fixes your problem. Regards, Jörg Höhle

1 0

Re: [iterate-devel] COMPILE FAILURE UNDER ALLEGRO CL 7
by Hoehle, Joerg-Cyril 13 Jun '05

13 Jun '05

Hi, Mark L. Williams CIV NSWC-PC wrote: >This is a strictly newbie question and I'm not on any mailing lists, so >please reply directly. > >I have downloaded the current version of iterate and have tried to >compile it with Allegro CL (Sparc version). Actually, I got it as part >of the the CL-pdf / CL-typesetting packages but have pulled it >independently because... Did you get cl-pdf and/or cl-typesetting to compile the iterate which is part of it? What version of iterate are you using? Please report something like file dates of the archive (not extraction date), since the file itself doesn't mention any precise enough version number. >the asdf build (executed by attempting to evaluate (asdf:operate >'asdf:load-op 'iterate) after establishing the necessary symbolic link >to the iterate.asd file) fails with the message: > >;;; Compiling file [path]/iterate.lisp >; While compiling (:top-level-form "iterate.lisp" 85868): >Error: DEFCLAUSE: required part of template (FOR var-spec &sequence) is >of odd length > >I've taken a quick look for similar-sounding bugs and/or any other hints >that might help, but I haven't found anything and I'm too new to >interpret and debug myself. Any help will be greatly appreciated. Never seen that myself. Are you using some beta version of Allegro-CL/sparc? Do you nkow about "modern" packages (i.e. case-sensitive reader)? Could some "modern" package get in the way? Can you try (apropos "&sequ") [or (apropos "&SEQU")]? How many symbols does it show? My suspicion is that there maybe 2 symbols of similar names, but distinct in case, and this may be causing trouble (i.e. &sequence and &SEQUENCE). Can you load, and/or compile the file by hand? (of course, need to load package.lisp initially) Can you load first, then compile? Can you compile in a fresh Lisp (not loaded first)? And finally, is there a problem with your mailer? I got your message as a text attachment within an attachment within an attachment. Maybe you should subscribe to iterate-devel for as long as your bug-report is unsolved, instead of hoping for some gentle moderator to forward your message to the list. iterate-devel has low traffic. Regards, Jörg Höhle

1 0

[iterate-devel] COMPILE FAILURE UNDER ALLEGRO CL 7
by Williams, Mark L CIV NSWC-PC 09 Jun '05

09 Jun '05

This is a strictly newbie question and I'm not on any mailing lists, so please reply directly. I have downloaded the current version of iterate and have tried to compile it with Allegro CL (Sparc version). Actually, I got it as part of the the CL-pdf / CL-typesetting packages but have pulled it independently because... the asdf build (executed by attempting to evaluate (asdf:operate 'asdf:load-op 'iterate) after establishing the necessary symbolic link to the iterate.asd file) fails with the message: ;;; Compiling file [path]/iterate.lisp ; While compiling (:top-level-form "iterate.lisp" 85868): Error: DEFCLAUSE: required part of template (FOR var-spec &sequence) is of odd length I've taken a quick look for similar-sounding bugs and/or any other hints that might help, but I haven't found anything and I'm too new to interpret and debug myself. Any help will be greatly appreciated. VR/ Mark L. Williams

1 0

Re: [iterate-devel] Version plan for Iterate 1.4 and 1.5
by Hoehle, Joerg-Cyril 02 Jun '05

02 Jun '05

Marc Battyani wrote: >> Plans for 1.5: >> + Move more code from the initially / loop prologue section >inside the let >> binding initialization, which might cause some user code to >fail because of code movement >Please don't wreck that! It's the reason I initially started >to use iterate. >Those parts (prologue/epilog/var bindings) are under (and >badly) specified >in LOOP and implementations differ on this. So let's keep it >in a sane way. What I mean is that Iterate is inconsistent. Did you know that WITH clauses get completely placed inside the surrouding let, while FOR ... ON <list> gets set in the :initially section, while other (e.g. reducing ...) also get set inside LET? I wanted to move as much as possible inside the surrouding LET. There's a *small* posibility that user code will break. In fact, I expect none of the current tests to fails as a result of such a change. >> For a farther future: move to another design for iteration >Do you already have some ideas on this ? No. gclcvs (which passed most Iterate tests) reminded me of the other Iterate package featuring gathering. I'll look at that. What really disturbs me with Iterate has a design weakness about referential transparency and inadvertent capture, e.g. (iter (with-some-other-macro (collect foo))) when with-some-other-macro itself expands to an iterate loop. It similar to those cases when an implicit block would change the target of a return statement. There's IMHO no way around that deficiency in Iterate. LOOP OTOH is robust against that, since it knows no clauses nested inside normal Lisp forms. Regards, Jörg.

1 0

[iterate-devel] Version plan for Iterate 1.4 and 1.5
by Hoehle, Joerg-Cyril 02 Jun '05

02 Jun '05

Hi, here's how I see the next steps w.r.t. Iterate. I've already sent a version of Iterate 1.4 to some of you (and it's on common-lisp.net thanks to Marco). I want 1.4 to be the last really compatibly-conservative Iterate around, before I'll make some possibly incompatible changes in 1.5. Plans for minor updates to 1.4: + some more bug fixes (e.g. did you know that in-stream did not work correctly with destructuring)? -- some already done (but not yet sent to you). + most important: additions to the documentation: - a glossary (what's the loop epilogue anyway?) - a reference of which clause is allowed at top-level only, as a generator etc. Plans for 1.5: + Move more code from the initially / loop prologue section inside the let binding initialization, which might cause some user code to fail because of code movement + harmonize "clause identifiers" across defmacro-driver, remove-clause and display-iterate-clause + possible change to thereis (different interaction with other clauses like always etc.) Somewhere in between: + other minor modifications (e.g. avoid duplication of code in some clauses) + deprecate add-loop-body-wrapper (a misnomer) and export add-iterate-wrapper or wrap-iterate-form or wrap-iterate-with wrap-iterate-using -- what do you suggest? + numerous other topics I have on my list For a farther future: move to another design for iteration that does not need a code-walker and as such suffers from the problematic of pre-order code traversal, or consider using a portable code walker library or implementation-dependent code. BTW, I just run Iterate through gclcvs (after delete-package "iterate" already present there) and it mostly worked, except for problems with macro-function, symbol-macrolet etc. Regards, Jorg Hohle.

2 1

Re: [iterate-devel] iterate 1.3?
by Hoehle, Joerg-Cyril 02 Jun '05

02 Jun '05

Hi, back in February I had promised: > + Give me time to write that review about *macroexpand-hook >and augment-environment. Now that I said so much about it, I >think I owe you all to write it up in detail. I realize I've now forgotten most of my analysis about why the cltl2 augment-environment is limited and doesn't scale to the large. The one thing I can remember was one concrete bug in Andreas Fuchs' code, independent on the weakness of the augment-environment design of CLtL2. Andreas binds *macroexpand-hook* to some handler of his own. Obviously, he does so because he needs the functionality. Correct working of his code depends on this binding. But if his code doesn't recognize an interesting form, he calls #'FUNCALL on the expander. That's bogus and won't work in the large. Consider another macro package of similar complexity, which also depends on rebinding *macroexpand-hook*. This package will fail mysteriouly because Andreas' code bypasses it. Or consider another Andreas' code expanding another macro that behave as badly: Andreas' handler will never get called, and Iterate will fail somehow. The only correct way for a handler is, instead of #'funcall, to invoke the previous value of *macroexpand-hook*. It's only such a weak form of "declining" that's available to the handler. BTW, CLHS says it as well: "Users who put their own function into *macroexpand-hook* should consider saving the previous value of the hook, and calling that value from their own." So that was the bug, and I'm again sorry I forgot most of my reflection about the limits of the augment-environment and brothers design. Regards, Jorg Hohle

1 0