[iterate-devel] Inaccurate *env* for Macroexpansions
Howdy List, I was having problems when using iterate with the arnesi call/cc codewalker, because variables bound by iterate were not in the environment of any macros in an iterate body form. The codewalker therefore gets that foo is an unbound variable in the iter clause below. To mitigate this, I patched iterate to update the *env* variable to reflect any bindings that iterate is going to make (in sbcl only ATM). Pulling some code from (bsd licenced) http://common-lisp.net/project/bese/repos/arnesi_dev/src/lexenv.lisp, I could probably write an implementation of the two methods I added in all the arnesi supported environments (lispworks /clisp /allegro /cmucl), but I have no way of testing these, so for now I stuck with a tested implementation for sbcl. The semantics of this are fairly straightforward, in that any variable declared textually before a macro call will appear as a lexical variable in the environment passed to the macroexpansion. (loop for foo upfrom 0 while (= foo 0) collect (funcall (lambda () (arnesi:with-call/cc foo)))) => (0) (iter (for foo upfrom 0) (while (= foo 0)) (collect (funcall (lambda () (arnesi:with-call/cc foo))))) => Before my patch this throws unbound variable foo exception / lexical environment is empty After my patch this "correctly" returns (0) Cheers, Russ New patches: [update environment with iterate produced variables Russ Tyndall <russ@acceleration.net>**20090520185521 Iterate produces variable bindings that are useful/necessary to macros expanding inside of an iterate body. This particularly effects other codewalkers (such as arnesi). We want (if possible) to update the env we expand macros in to reflect any variables that iterate is lexically binding. ] { hunk ./iterate.lisp 655 +(defgeneric lexical-variables (environment) + (:documentation "Return the names of all the local variables + in ENVIRONMENT. Does not return neither symbol-macrolets nor + ignared variables.") + (:method ((env T)) ())) + +#+sbcl +(defmethod lexical-variables ((environment sb-kernel:lexenv)) + (loop for var-spec in (sb-c::lexenv-vars environment) + when (and (atom (cdr var-spec)) + (not (and (typep (cdr var-spec) 'sb-c::lambda-var) + (sb-c::lambda-var-ignorep (cdr var-spec))))) + collect (car var-spec))) + +(defgeneric update-environment (env) + (:method ((env T)) env) + (:documentation "Updates the lexical environment to reflect all of the available + bindings that iterate has thus far created ")) + +#+sbcl +(defmethod update-environment ((env sb-kernel:lexenv)) + (let ((new-bindings + (loop for (sym . _) in *bindings* + unless (or (null (symbol-package sym)) + (find sym (lexical-variables *env*))) + collect (cons sym T)))) + (sb-c::make-lexenv :default env :vars new-bindings))) + hunk ./iterate.lisp 709 + + ;; Iterate produces variable bindings that are useful/necessary to macros expanding inside + ;; of an iterate body. This particularly effects other codewalkers (such as arnesi). + ;; We want (if possible) to update our environment to reflect any variables that + ;; iterate is lexically binding + (setf *env* (update-environment *env*)) + } Context: [first-time-p bugfix: return-code :body must return list of forms Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525141533 if-first-time not declared obsolete documentation strings for (iter:display-iterate-clauses) complete ] [fix defmacro-driver example in manual Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525081443] [Use @:, @. and two spaces between sentences Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525080932 Move section on predicate (first-time-p) outside of gathering clauses Various typos and some clarifications ] [document *list-end-test* removal in FOR...IN+ON Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525074338] [Renamed back to sharpL-reader attila.lendvai@gmail.com**20070506100744] [Fix sharpL reader, add :execute to the eval-when to make (load "iterate" :compiling t) work on clisp attila.lendvai@gmail.com**20070506100704] [Convert manual to Texinfo. Luis Oliveira <loliveira@common-lisp.net>**20060713142915] [make FOR...IN/ON with dotted lists work like LOOP hoehle@users.sourceforge.net**20070503130604 More precisely, FOR ON accepts dotted lists, FOR IN errors out. As a result, iterate::*list-end-test* was eliminated. Behaviour is now constant and does not depend on some special variable. Note: Documentation not yet updated, pending move to Texinfo. ] [walk-tagbody: more testcases Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070503095309] [walk-tagbody must not macroexpand symbol/tags among its statements Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070404124132] [add ELSE test cases, remove GNU Arch tag Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070503093008] [Clean up #L stuff, do not leave #L enabled after loading iterate attila.lendvai@gmail.com**20070426153431] [Set *list-end-test* to 'endp instead of 'atom, so (iter (for foo :in something-non-list)) fails instead of silently exists attila.lendvai@gmail.com**20070215151652] [wrap code in progns, to avoid possiblity of multiple nil tags in tagbody Henrik Hjelte <henrik@evahjelte.com>**20061025145324] [test to detect bug, more than one nil tag in tagbody Henrik Hjelte <henrik@evahjelte.com>**20061025145128] [Added release.sh attila.lendvai@gmail.com**20060506155953] [TAG 1.4.3 attila.lendvai@gmail.com**20060505134701] Patch bundle hash: 62134238d49a98e5dbfb86589229ec0896284f3d
I just noticed that the same problem was occuring with locally declared variables, but that iterate was tracking these in the *internal-variables* variable. Therefore I added a second patch that updates the environment with these as well. Cheers, Russ New patches: [update environment with iterate produced variables Russ Tyndall <russ@acceleration.net>**20090520185521 Iterate produces variable bindings that are useful/necessary to macros expanding inside of an iterate body. This particularly effects other codewalkers (such as arnesi). We want (if possible) to update the env we expand macros in to reflect any variables that iterate is lexically binding. ] { hunk ./iterate.lisp 655 +(defgeneric lexical-variables (environment) + (:documentation "Return the names of all the local variables + in ENVIRONMENT. Does not return neither symbol-macrolets nor + ignared variables.") + (:method ((env T)) ())) + +#+sbcl +(defmethod lexical-variables ((environment sb-kernel:lexenv)) + (loop for var-spec in (sb-c::lexenv-vars environment) + when (and (atom (cdr var-spec)) + (not (and (typep (cdr var-spec) 'sb-c::lambda-var) + (sb-c::lambda-var-ignorep (cdr var-spec))))) + collect (car var-spec))) + +(defgeneric update-environment (env) + (:method ((env T)) env) + (:documentation "Updates the lexical environment to reflect all of the available + bindings that iterate has thus far created ")) + +#+sbcl +(defmethod update-environment ((env sb-kernel:lexenv)) + (let ((new-bindings + (loop for (sym . _) in *bindings* + unless (or (null (symbol-package sym)) + (find sym (lexical-variables *env*))) + collect (cons sym T)))) + (sb-c::make-lexenv :default env :vars new-bindings))) + hunk ./iterate.lisp 709 + + ;; Iterate produces variable bindings that are useful/necessary to macros expanding inside + ;; of an iterate body. This particularly effects other codewalkers (such as arnesi). + ;; We want (if possible) to update our environment to reflect any variables that + ;; iterate is lexically binding + (setf *env* (update-environment *env*)) + } [makes sure that *internal-variables* are taken into account in the macroexpansion environment as well as *bindings* Russ Tyndall <russ@acceleration.net>**20090520193752] { hunk ./iterate.lisp 677 - (loop for (sym . _) in *bindings* + (loop for sym in (append (mapcar #'car *bindings*) *internal-variables*) hunk ./iterate.lisp 714 - (setf *env* (update-environment *env*)) + (let ((*env* (update-environment *env*))) hunk ./iterate.lisp 716 - ;; Some compilers (e.g. Lucid on Sparcs) treat macros differently at - ;; compile-time; macroexpand does not expand them. We assume that if - ;; this happens, macroexpand's second value is nil. - ;; What do we do with the form in that case? This is actually a - ;; very serious problem: if we don't walk it, we miss things, but if we - ;; do walk it, we don't know how to walk it. Right now, we don't walk - ;; it and print out a warning. - ;; --Jeff Siskind says try binding *macroexpand-hook* to #'funcall. - (multiple-value-bind (ex-form expanded?) - (macroexpand-1 form *env*) - (cond - (expanded? (walk ex-form)) - (t (clause-warning "The form ~a is a macro that won't expand. ~ + ;; Some compilers (e.g. Lucid on Sparcs) treat macros differently at + ;; compile-time; macroexpand does not expand them. We assume that if + ;; this happens, macroexpand's second value is nil. + ;; What do we do with the form in that case? This is actually a + ;; very serious problem: if we don't walk it, we miss things, but if we + ;; do walk it, we don't know how to walk it. Right now, we don't walk + ;; it and print out a warning. + ;; --Jeff Siskind says try binding *macroexpand-hook* to #'funcall. + (multiple-value-bind (ex-form expanded?) + (macroexpand-1 form *env*) + (cond + (expanded? (walk ex-form)) + (t (clause-warning "The form ~a is a macro that won't expand. ~ hunk ./iterate.lisp 732 - (list form))))) + (list form)))))) } Context: [first-time-p bugfix: return-code :body must return list of forms Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525141533 if-first-time not declared obsolete documentation strings for (iter:display-iterate-clauses) complete ] [fix defmacro-driver example in manual Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525081443] [Use @:, @. and two spaces between sentences Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525080932 Move section on predicate (first-time-p) outside of gathering clauses Various typos and some clarifications ] [document *list-end-test* removal in FOR...IN+ON Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070525074338] [Renamed back to sharpL-reader attila.lendvai@gmail.com**20070506100744] [Fix sharpL reader, add :execute to the eval-when to make (load "iterate" :compiling t) work on clisp attila.lendvai@gmail.com**20070506100704] [Convert manual to Texinfo. Luis Oliveira <loliveira@common-lisp.net>**20060713142915] [make FOR...IN/ON with dotted lists work like LOOP hoehle@users.sourceforge.net**20070503130604 More precisely, FOR ON accepts dotted lists, FOR IN errors out. As a result, iterate::*list-end-test* was eliminated. Behaviour is now constant and does not depend on some special variable. Note: Documentation not yet updated, pending move to Texinfo. ] [walk-tagbody: more testcases Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070503095309] [walk-tagbody must not macroexpand symbol/tags among its statements Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070404124132] [add ELSE test cases, remove GNU Arch tag Joerg-Cyril Hoehle <hoehle@users.sourceforge.net>**20070503093008] [Clean up #L stuff, do not leave #L enabled after loading iterate attila.lendvai@gmail.com**20070426153431] [Set *list-end-test* to 'endp instead of 'atom, so (iter (for foo :in something-non-list)) fails instead of silently exists attila.lendvai@gmail.com**20070215151652] [wrap code in progns, to avoid possiblity of multiple nil tags in tagbody Henrik Hjelte <henrik@evahjelte.com>**20061025145324] [test to detect bug, more than one nil tag in tagbody Henrik Hjelte <henrik@evahjelte.com>**20061025145128] [Added release.sh attila.lendvai@gmail.com**20060506155953] [TAG 1.4.3 attila.lendvai@gmail.com**20060505134701] Patch bundle hash: ae2d5b63f8a86b784836161897a5dea4164cb94b
participants (1)
-
Russ Tyndall