November 2004 - iterate-devel - mailman3.common-lisp.net

[iterate-devel] Macroexpansion, bindings and information flow (Iterate package)
by Hoehle, Joerg-Cyril 15 Nov '04

15 Nov '04

Hi, you're invited to look up comp.lang.lisp on this subject where I wrote: "I believe in its over 10 years of existence, it's never been able to correctly handle this form: (macrolet ((over(x) `(collect ,x))) (iterate (for n in '(1 2 3)) (flet ((over(x)(declare (ignore x)) (collect 2))) (over n)))) ; would yield (2 2 2) if correct One reason is that it does not contain a code walker aware of lexical bindings. ..." [BTW, I added that to iterate-test.lisp :)] I believe it's possible to add restricted support for macrolet. The restriction is that the macro expanders neither contain nor expand into iterate clauses (the body may contain clauses). This would mean that Iterate would play nice with uses of macrolet which are independent on it, e.g. hashtable iterators. Together with support for locally (and fixing some more subtle bugs I did not yet talk about), I believe Iterate would be more valuable if a) it accepted macrolet instead of giving an error b) there would be some restrictions b2) Iterate would issue (warn ...) upon macrolet c) the above shows that it is hard to have Iterate generate correct code in all cases, but users can hopefully live with that. My fear is that the Iterate user may not realize that some situations are deadly. E.g. I've often thought about an implementation style as follows: (defmacro loop-finish () (error "Use of LOOP-FINISH outside LOOP") ;; or `(error ...)) (defmacro loop (&body) `(macrolet (loop-finish () `(go loop-end)) ...)) The idea is to provide both global and local macro definitions. The global one is solely to ease debugging. Iterate would generate incorrect code when this loop definition would be used somewhere inside the iterate body. A real problem is that the user may not even notice that this situation arises. What do you think? Notice that correct macrolet handling requires correct lexical parsing. In my posting to comp.lang.lisp I'm asking how/whether this can be portably done. Regards, Jörg Höhle.

2 1

Re: [iterate-devel] Re: Macroexpansion, bindings and information flow (Iterate package)
by Hoehle, Joerg-Cyril 15 Nov '04

15 Nov '04

Andreas Fuchs wrote: >> I believe in its over 10 years of existence, it's never been able to >> correctly handle this form: >> (macrolet ((over(x) `(collect ,x))) >> (iterate >> (for n in '(1 2 3)) >> (flet ((over(x)(declare (ignore x)) (collect 2))) >> (over n)))) ; would yield (2 2 2) if correct >Actually, the current version of iterate can handle that clause pretty >well Did you try it out? > (since iterate--main--1.0--patch-6, since when we pass the *env* >argument to macroexpand calls). I don't have GnuArch so I don't don't what this version refers to, but my copy of iterate-1.0.9 has *env*, &environment and two-argument calls to macroexpand. I claim that it does not handle it because Iterate will apply the MACROLET expander to the code, because it ignores the shadowing function definition. The problem is independent on macrolet. defmacro would yield the same behaviour. Does your copy of Iterate really return (2 2 2) ?!? >It seems like most implementations offer code walker support similar >to ClTl2's. While not ANSI CL, these might be your best bet. What are your thoughts w.r.t. a release of Iterate which is not portable, but customized to a restricted set of implementations (that we could put our hands on -- I only have CLISP and Corman Lisp here ...)? -- supposing that all interesting implementations export their code walkers in a useable way. Regards, Jörg Höhle

2 1

[iterate-devel] value of the top-level restriction?
by Hoehle, Joerg-Cyril 15 Nov '04

15 Nov '04

Hi, why are some forms restricted to top-level, while others are not? For example, the following looks like a good replacement of FOR IN which is not restricted to top-level iteration: (iter (repeat 5) (let () ; not at top-level (with l = '(1 2 3)) (for el do-next (if (null l) (terminate) (dsetq el (pop l)))) (princ (cons "hi" el)))) (hi . 1)(hi . 2)(hi . 3) This could be wrapped via iterate:defxyz and even be used as a generator. Likewise, the in-whole-vector example in the manual IMHO provides a better iterator than the built-in IN-VECTOR: it is not restricted to top-level -- it uses FOR DO-NEXT. (iterate(repeat 5)(let()(for x in-vector #(1 2 3))(princ x))) Iterate, in (FOR X IN-VECTOR #(1 2 3)): Clause can occur only at top-level [Condition of type SIMPLE-ERROR] Does anybody know the reason of the top-level check? Is it a kind of user safeguard? Should the meaning of "TOP-LEVEL" in Iterate be redefined? Regards, Jörg Höhle.

1 0

[iterate-devel] towards cl:locally support in Iterate
by Hoehle, Joerg-Cyril 15 Nov '04

15 Nov '04

Hi, J.amsterdam wrote: ;;; - Consider allowing (locally...). The semantics would be quite ;;; odd, however. I have some code for locally, but there's one thing open: A) Should the forms therein be considered top-level or not w.r.t. Iterate? ANSI-CL processes locally at top-level, in a possibly non-null lexical environment... That's ANSI, but here we care about Iterate: shall it accept clauses like (FOR ...) which are otherwise limited to Iterate's top-level? My implementation is currently as follows: locally is walked with declarations therein ignored. Sorry, that's already too vague... What do you think? (iterate (repeat 10) (locally (declare whatever) (for i in '(1 2 3)) (declare (fixnum i)))) B1) allow top-level clause or not? B2) if allowed: recognize/create global fixnum declaration or not? not how (declare fixnum) is not at the right place for locally, but at a typical place of iterate clauses. Sorry, maybe this is too confusing. Maybe I should just go ahead and implement something, and you tell me if you're happy. The issue is somewhat similar to macrolet and lexical scoping: there could be cases where the code is incorrect, yet Iterate could still be useful -- "good enough for daily use". Possible incorrectness is not quite the environment that I appreciate, but... BTW, I'd appreciate if somebody could come up with an example of a non-top-level clause endangered by use of locally. I'm looking for something different from the trivial (declare special) corruption. C) Would you mind if B2) differed from (iterate (repeat 10) (locally (declare whatever) (declare (fixnum i)) (for i in '(1 2 3)))) or would you take this to conclude C2) that every declaration information from within locally shall be thrown away, not just the first ones? C3) Top-level inside locally is bad anyway? Regards, Jörg Höhle.

1 0

[iterate-devel] Re: Macroexpansion, bindings and information flow (Iterate package)
by Andreas Fuchs 15 Nov '04

15 Nov '04

(This is the first time I've opened c.l.l in a long time, and I didn't intend to post anything, but then... (-:) I'm cross-posting this to the iterate-devel mailing list. On Mon, 15 Nov 2004 10:15:35 +0100, Joerg Hoehle wrote: > I came across J.Amsterdam's Iterate package and would like to address > its main weakness: correct, lexical aware code expansion. > > I believe in its over 10 years of existence, it's never been able to > correctly handle this form: > (macrolet ((over(x) `(collect ,x))) > (iterate > (for n in '(1 2 3)) > (flet ((over(x)(declare (ignore x)) (collect 2))) > (over n)))) ; would yield (2 2 2) if correct Actually, the current version of iterate can handle that clause pretty well (since iterate--main--1.0--patch-6, since when we pass the *env* argument to macroexpand calls). What it can't handle (and what I think you meant to post, anyway) are nested macrolet clauses, like: (iterate (macrolet ((over(x) `(collect ,x))) (for n in '(1 2 3)) (flet ((over(x)(declare (ignore x)) (collect 2))) (over n)))) ; would yield (2 2 2) if correct because the code walker does not know how to extend the environment with this nested macrolet. > One reason is that it does not contain a code walker aware of lexical > bindings. Right. I think that's partly due to the fact that it's impossible to have a macrolet (and symbol-macrolet)-aware code walker in portable CL without reimplementing half of common lisp. (-: > I actually wonder whether Iterate actually needs a full code walker, > thereby duplicating functionality found in every Lisp implementation, > or whether Iterate could be written without one, possibly by clever > use of MACROLET and/or *MACROEXPAND-HOOK*? An iterate that doesn't code walk could end up being more friendly to its maintainer. Iterate in its current form relies very heavily on the code walker, so you'd end up re-implementing most of it from scratch (probably not such a bad thing, either (-:) *macroexpand-hook*, now there's an idea. From the code walker, it might be possible to put a closure in there which can expand the macrolet by itself. That would be interesting to try out (you'd still have to create a macro function from the macrolet's definition. hmm) > Here's one of the problems: Iterate allows a (COLLECT x INTO var) > clause to appear anywhere deep inside the body of (Iterate ...), > contrary to LOOP. As a result, it's only after COLLECT has been > encountered that Iterate knows which bindings it must create in its > top level expansion. Information flows from deep inside to the > outermost level. How to do that? > > E.g. > (iterate ... (collect x into a) ... (collect ... into b) ...) > -> `(let (a b) ... ...) > (iterate ... no-collect-at-all ...) > -> `(let () ...) or (progn ...) > > Do you have an idea how such a COLLECT INTO macro could be written > without duplicating a code walker? Unfortunately, ANSI CL doesn't specify the order in which macros are expanded, but if it did, you /might/ be able to signal conditions which the outer *macroexpand-hook* functions could know how to treat. All this is highly speculative, but it would be a pretty neat hack. (-: > This seems hairy to me if one wants it to work both in interpreted > and compiled modes, since with an interpreter, the COLLECT form may > only be encountered upon evaluation of (COLLECT #) --really late. Right. That's probably part of the reason why ANSI doesn't specify macroexpand order. > If it's not possible without a code walker, are there portable means > to use the implementation's code walking engine? It seems a pity to me > that something like AUGMENT-ENVIRONMENT (in CLtL2) was not > standardized in ANSI-CL. > Section 8.5, Environments, of CLtL2 see > http://www.supelec.fr/docs/cltl/clm/node102.html It seems like most implementations offer code walker support similar to ClTl2's. While not ANSI CL, these might be your best bet. Cheers, -- Andreas Fuchs, <asf(a)boinkor.net>, asf(a)jabber.at, antifuchs

1 0

[iterate-devel] Jorg's iterate patches against 1.0.9
by Hoehle, Joerg-Cyril 15 Nov '04

15 Nov '04

Hi, here's my current set of patches against iterate-1.0.9 One thing I want to mention here is to bump the version number. I believe J.Amsterdam last was 1.2, so the 200X version should start with 1.3. I did so in the source. The diffs are in -u3 format, and if Andreas perfers small increments, I've included my RCS ,v file. Andreas, are you going to put up a new release based on that? If you do so, I'll base my next patches upon it. What I did not include so far is o changed order of special-form/macro recognition. Are there any more arguments to debate? o walk-locally (currently not quite correct) o some restricted macrolet support o some more manual clarifications Here's a mini-Changelog o package.lisp + use declaim + use (export list): (export +clause-names+ '#:iterate) + export defsynonym defmacro-driver o iterate.lisp + (declaim (declaration declare-variables)) + listp -> consp + (prog1 (car body) (pop body)) -> (pop body) + (code-char 0) is portable, #\nul or #\null is not + counting uses '(subst (var exp) #) instead of expr + bang-vars-1: tail-call optimized o for in-package(s) evaluates package(s) argument o for in-package(s) work as generators + use #:start/beginning to (hopefully) work with Allegro modern style + walk-flet: accepts declarations + remove walk-identity and use (x . nil) consistently in *special-form-alist* o iter-man.tex + drop note about future with-hashtable/package-iterator o iter-pg.lisp + multiple-value-prog1 + avoid zerop length on lists ; ---------------- DONE o minor CL optimizations: consp, pop, export, declaim o for in-package(s) evaluates package(s) argument o for in-package(s) works as generator o for in-hashtable usable as generator o for in-hashtable supports destructuring o for in-packages was broken because of 'pkgs symbol-types are not evaluated in with-package-iterator and must not be empty => consp o (for vars DO-NEXT) now binds vars o use dsetq instead of destructuring-bind for consistency of templates o a few more package exports o added more testcases o (dsetq (values x nil y)) works Regards, Jörg Höhle.

1 0

[iterate-devel] for i in-vector downto 0 by 2 yields what?
by Hoehle, Joerg-Cyril 12 Nov '04

12 Nov '04

Hi, I'm adding a few test cases to iterate-test.lisp. I wonder about the interaction of in-vector and by when moving backwards. ITER> (iter (for i index-of-vector #(0 1 2 3 4) downto 0 by 2) (collect i)) (3 1) I'd have expected (4 2 0) here. What do you think? By analogy: ITER> (iter(for x downfrom 4 by 2)(repeat 3)(collect x)) (4 2 0) ITER> (loop for x downfrom 4 by 2 repeat 3 collect x) (4 2 0) ITER> (iter (for i index-of-vector #(0 1 2 3 4) downto 0 by 1) (collect i)) (4 3 2 1 0) That is fine with me. I'd expect behaviour like (iter (for x rom (1- (length array)) downto 0 [by N]) (collect x)) (4 2 0) ; when by 2 Rationale: the first index/element it the rightmost, then walk by N. The current behaviour looks to me like an unintended side effect of the implementation, moving down from (length array)=5 by 2 to yield 3,1. Regards, Jörg Höhle.

3 2

[iterate-devel] walk-special-forms first (was Iterate broken on A CL 7.0 beta?)
by Hoehle, Joerg-Cyril 12 Nov '04

12 Nov '04

Hi, walking through the archives, I came across this thread. Indeed, I believe Allegro 7.0 to be broken to macroexpand (cond ...) into (cond ...) again. But Iterate-1.0.9 is IMHO broken as well. I recommend doing the following steps: 1. if operator is in *special-form-alist*, call handling function 2. if macro-function, tail-recursively expand it 3. if still special-operator-p, complain (and possibly default to function call) 4. if clause, handle it 5. else function call Rationale: The argument "To make maintenance of ITERATE easier, I prefer expanded macros over special forms that I have to treat specially." is IMHO moot. Iterate MUST recognize and handle the known special forms, because that's what can come out of macroexpansion. So it may as well do it immediately. This has the additional benefit that it can handle these consistently across platforms. For the sake of Allegro, walk-cond form must remain, otherwise it'll loop endlessly. I expect no other platform specific hacks. My claim of "portable behaviour" is backed by this observation: In CLISP, declare behaves as a macro and expands to NIL. I can't judge whether this is bogus, weird or not. As a result, walk-declare is never called. This explains why declarations are not kept in the code as the manual claims when using CLISP with iterate-1.0.9. Only when Iterate treats known special forms first can it guarantee that its walkers will be used. Changing the order to do so makes declarations show up in CLISP. Handling known special operators first is what iterate-1.0.lisp and -1.2.lisp did. Steps 3 and 4 are interchangeable, as a matter of taste. Regard, Jörg Höhle

1 0

[iterate-devel] iterate package source code server
by Hoehle, Joerg-Cyril 12 Nov '04

12 Nov '04

Hi, it's a great idea to put valuable but old packages from the AI repository or whereever to an open source server on the net! I've done several changes that I'll release soon, but there are are a few points I'd like to discuss before: o Participation: GNUArch to me sounds like yet another CVS system. It causes me overhead as I have to install yet anoter SW on the PC. CVS, subversions, GNUArch, where will it end? Is that really necessary? o "to quote or not to quote" -- see separate e-mail. Here's a list of changes which I've done so far: o minor CL optimizations: consp, pop, export, declaim o for in-package(s) evaluates package(s) argument o for in-package(s) works as generator o for in-hashtable usable as generator o for in-hashtable supports destructuring o for in-packages was broken because of 'pkgs symbol-types are not evaluated in with-package-iterator and must not be empty => consp o (for vars DO-NEXT) now binds vars o use dsetq instead of destructuring-bind for consistency of templates + (code-char 0) is portable, #\nul or #\null is not + use #:start/beginning to (hopefully) work with Allegro modern style + tiny changes to manual o added more testcases Planned: support (dsetq (values x nil y) (foo)) to ignore second value Regards, Jörg Höhle.

2 1

[iterate-devel] iterate proposal: "maybe quote or not" is bad design: remove from manual
by Hoehle, Joerg-Cyril 11 Nov '04

11 Nov '04

Hi, I'd like to change the few places where the manual says "may be quoted, but need not be". Rationale: don't make the user believe something is evaluated when it's not! Despite the manual says so, (collect i at 'beginning) barfs. Instead of changing the code to make this work, I propose to change the manual to *not* accept quoted forms. 1. These confuse the user. They fool the user into believing arbitrary forms could be used (if (foo) 'beginning 'end) and that it's a run-time choice when it's not. 2. These make implementations contain broken heuristics like (and consp (eq car 'quote)) I believe there's more value in making such forms accept various forms of symbols (like loop keywords), e.g. :end, end END etc. instead. On a similar issue, I have criticized UFFI for containing similar heuristics instead of clear descriptions of what types are evaluated and what not (cf. uffi-users list of October 2004). Kevin Rosenberg is about to change something there. Cost of change to users: it's a pure matter of style. Users using 'end/'beginning/'start need to use other forms, e.g. (collect foo :at :start) (collect foo at :start) (collect foo :at #:start) (collect foo #:at start) etc. All of these can be supported cheaply (in terms of run-time) and I'm sure all users will find a form that pleases them visually. Prior to making these changes in (my copy of) the code, I wanted to discuss the issue at first. Regards, Jörg Höhle

1 0