- iterate-devel - mailman3.common-lisp.net

[iterate-devel] Version plan for Iterate 1.4 and 1.5
by Hoehle, Joerg-Cyril 02 Jun '05

02 Jun '05

Hi, here's how I see the next steps w.r.t. Iterate. I've already sent a version of Iterate 1.4 to some of you (and it's on common-lisp.net thanks to Marco). I want 1.4 to be the last really compatibly-conservative Iterate around, before I'll make some possibly incompatible changes in 1.5. Plans for minor updates to 1.4: + some more bug fixes (e.g. did you know that in-stream did not work correctly with destructuring)? -- some already done (but not yet sent to you). + most important: additions to the documentation: - a glossary (what's the loop epilogue anyway?) - a reference of which clause is allowed at top-level only, as a generator etc. Plans for 1.5: + Move more code from the initially / loop prologue section inside the let binding initialization, which might cause some user code to fail because of code movement + harmonize "clause identifiers" across defmacro-driver, remove-clause and display-iterate-clause + possible change to thereis (different interaction with other clauses like always etc.) Somewhere in between: + other minor modifications (e.g. avoid duplication of code in some clauses) + deprecate add-loop-body-wrapper (a misnomer) and export add-iterate-wrapper or wrap-iterate-form or wrap-iterate-with wrap-iterate-using -- what do you suggest? + numerous other topics I have on my list For a farther future: move to another design for iteration that does not need a code-walker and as such suffers from the problematic of pre-order code traversal, or consider using a portable code walker library or implementation-dependent code. BTW, I just run Iterate through gclcvs (after delete-package "iterate" already present there) and it mostly worked, except for problems with macro-function, symbol-macrolet etc. Regards, Jorg Hohle.

2 1

Re: [iterate-devel] iterate 1.3?
by Hoehle, Joerg-Cyril 02 Jun '05

02 Jun '05

Hi, back in February I had promised: > + Give me time to write that review about *macroexpand-hook >and augment-environment. Now that I said so much about it, I >think I owe you all to write it up in detail. I realize I've now forgotten most of my analysis about why the cltl2 augment-environment is limited and doesn't scale to the large. The one thing I can remember was one concrete bug in Andreas Fuchs' code, independent on the weakness of the augment-environment design of CLtL2. Andreas binds *macroexpand-hook* to some handler of his own. Obviously, he does so because he needs the functionality. Correct working of his code depends on this binding. But if his code doesn't recognize an interesting form, he calls #'FUNCALL on the expander. That's bogus and won't work in the large. Consider another macro package of similar complexity, which also depends on rebinding *macroexpand-hook*. This package will fail mysteriouly because Andreas' code bypasses it. Or consider another Andreas' code expanding another macro that behave as badly: Andreas' handler will never get called, and Iterate will fail somehow. The only correct way for a handler is, instead of #'funcall, to invoke the previous value of *macroexpand-hook*. It's only such a weak form of "declining" that's available to the handler. BTW, CLHS says it as well: "Users who put their own function into *macroexpand-hook* should consider saving the previous value of the hook, and calling that value from their own." So that was the bug, and I'm again sorry I forgot most of my reflection about the limits of the augment-environment and brothers design. Regards, Jorg Hohle

1 0

Re: [iterate-devel] devel access?
by Hoehle, Joerg-Cyril 20 May '05

20 May '05

Hi, this is a short note that I'm making progress towards better net connectivity despite corporate firewalls. So far, I've accessed: o cvs-ssh.suorceforge.net (because they redirect port 80+443 to the ssh port and I have developer access there, but not yet managed to migrate my key from F-Secure (commercial) SSH to Ubuntu/Debian's Openssh on the laptop. o cvs-pserver.sourceforge.net (redirect 80+443 to 2401) o common-lisp.net:iterate because Arch (baz) works via http I couldn't get to common-lisp.net:slime because the CVS (2401) port is not accessible. I started browsing through the Arch/baz docs and tutorials, but boy, I feel there are too many commands out there. Since Ubuntu/Debian is such a nice setting, I installed cmucl and sbcl (and gcl) packages (and slime-1.2.1 promptly failed for sbcl and clisp :-( Why did nobody tell me before that both cmucl and sbcl have some failing iterate tests? Now I believe I need to generate a gpg key (instead of pgp ones) to get write access to common-lisp.net? Regards, Jörg Höhle.

1 0

Re: [iterate-devel] type declarations from the Iterate package
by Hoehle, Joerg-Cyril 29 Mar '05

29 Mar '05

Hi, Marc Battyani wrote: [Thanks for your feedback] >> I was wondering (given zero e-mail traffic on this subject here) if >>somebody had noticed that Iterate outputs quite dubious type declarations >when being asked for them (i.e. (declare (iterate:declare-variables)))? >I used it a few times. The (declare (iterate:declare-variables)). > IIRC it worked for me. I always tend to express myself too generally :-( For *some* constructs, Iterate outputs wrong type declarations. It depends on a) the construct and b) the implementation (more precisely the results of TYPE-OF). E.g. in CLISP, (type-of 0) -> BIT and (macroexpand-1'(iter:iter(declare(iter:declare-variables))(iter:repeat 0))) reveals (LET* ((#:COUNT3 0)) (DECLARE (TYPE BIT #:COUNT3)) ...) whose type is evidently broken. Obviously the code was written with (type-of 0) -> FIXNUM in mind. Widening that topic, I'm wondering whether generally, even the implementation's LOOP or other iteration constructs always produce correct declarations in border cases. E.g. suppose array-dimension-limit = most-positive-fixnum Loop code traversing a vector could possibly be written like: (declare (fixnum running-var)) ; because array-dimension fits a fixnum (incf running-var step) ; might escape fixnum limit (when (> running-var limit) (loop-finish)) but not like this (let ((var (+ running-var step))) ; no type declaration for var, ; could be a bignum after last iteration (if (> var limit) (loop-finish)) (setq running-var var)) ; in this case var is not too large Similarly, consider (dotimes (i most-positive-fixnum) (declare (fixnum i)) ...) Within the user-visible body, the type declaration holds. Yet dotimes could be implemented using (incf i) (when (> i #:limit) ...), thereby breaking the type declaration. Is that just academic and I'm worrying too much? Regards, Jörg Höhle.

1 0

[iterate-devel] type declarations from the Iterate package
by Hoehle, Joerg-Cyril 18 Mar '05

18 Mar '05

Hi, I was wondering (given zero e-mail traffic on this subject here) if somebody had noticed that Iterate outputs quite dubious type declarations when being asked for them (i.e. (declare (iterate:declare-variables)))? On CLISP, nobody notices, since they are ignored anyway, but on CMUCL... Did anybody understand the *intent* of J. Amsterdams type handling explained in the Iterate manual? 1. It looks to me that Iterate's handling of type declarations is somewhat broken, at least in the light of current compiler standards (things looked differently ten years ago). Other than that, I fixed a few unrelated bugs and will soon export remove-clause -- don't hold your breath. After that, I'll try to look into the topic of type declarations. Does anybody care about these at all anyway?? Regards, Jorg Hohle

2 1

[iterate-devel] Integration with CL-ENUMERATIONS and ITERATE
by Hoehle, Joerg-Cyril 18 Mar '05

18 Mar '05

Hi, I wrote: >It appears that it [iterate for Postgresql result sets] > cannot currently be written using only names exported >from the ITERATE package. The only thing lacking is an >interface to add-loop-body-wrapper Well, I find add-look-body-wrapper quite an odd name to be exported from the Iterate package. OTOH, I'm quite against gratuitous name changes to software packages that have been basically working for the last ten years. So far, I have (hopefully) managed to maintain strict compatibility. Should I a) just export iterate:add-loop-body-wrapper b) keep iterate::add-loop-body-wrapper and export a different name for the same function, e.g. create iterate:add-wrapper or iterate:add-wrapping-form c) maintain status quo What do users think? In case you don't remember, the issue at hand is that unless some such functionality is exported, one cannot pretend that Iterate presents an API (visible from the list of exported names) that can be used for extending Iterate in some useful ways (e.g. pretend FOR ... IN-FILE did not exist and provide it). Regards, Jorg Hohle

1 0

Re: [iterate-devel] Integration with CL-ENUMERATIONS and ITERATE
by Hoehle, Joerg-Cyril 11 Mar '05

11 Mar '05

Marco Antoniotti wrote: >I would like to provide an ITERATE extension to leverage the >ENUMERATION instances. I went through the manual but I am a little >unclear about how to user DEFMACRO-DRIVER etc. >Could anybody help me? Sure, send this to the list. I admit that I'm also confused by the proliferation of terms and lack of definitions (or hard to find definitions in flowtext), e.g. defclause defclause-driver defmacro-driver defmacro-clause defclause-sequence What's a clause? what's driver? etc. My finding so far is that a driver comes into play when FOR can be replaced by GENERATE. defmacro-* is best suited for users writing extensions, drawing from the macro concept, i.e. the body expands to other iterate clauses and CL forms. C.f. iterate-test.lisp I'll make a note that the manual should provide a glossary, or at least make the index work -- where does a TeX .ind file come from? Also unclear to me is top-level vs. anywhere forms. Why are some clauses (mostly FOR, but not all of them) restricted to the top-level? As an exercise, I just looked at iterate-pg.lisp. It appears that it cannot currently be written using only names exported from the ITERATE package. The only thing lacking is an interface to add-loop-body-wrapper for the sake of with-pg-cursor. Other than that, it could be written like (defmacro-driver (FOR var-spec IN-RELATION query ON-CONNECTION conn) (add-loop-body-wrapper `(with-pg-cursor ,cursor ,conn ,query)) `(for ,var-spec = (first (or (pg-result #) (finish)))) or `(for/generate ,var-spec next (first (or (pg-result #) (terminate)))) ) Alternatively, with-pg-cursor might be "inlined" into the expansion using iterate:finally-protected and with clauses. That would also prevent double evaluation of CONN. I recommend that you stick to the manual as long as possible and write a macro-clause or -driver. Don't look at the internals, don't use return-driver-code, make-binding etc. Actually, it may be a worthwhile exercise to rewrite some part of Iterate in a system construction manner, building drivers on top of others. It could make code shorter and possibly easier to maintain. E.g. FOR ... IN could be written as follows: (defmacro-driver (FOR var IN list &optional BY (step ''cdr)) (let ((for/gen (if generate 'generate 'FOR))) `(,for/gen (,var) ON ,list BY ,step))) and the expansion is only slightly worse than now (one temporary let) Regards, Jorg Hohle

1 0

[iterate-devel] Integration with CL-ENUMERATIONS and ITERATE
by Marco Antoniotti 10 Mar '05

10 Mar '05

Hi I just submitted a new package to common-lisp.net called CL-ENUMERATION. It is a rendition of the Java Enumeration/Iterator interfaces in CL. It is essentially a way to let Java, C++, Python etc programmer migrate to CL. I would like to provide an ITERATE extension to leverage the ENUMERATION instances. I went through the manual but I am a little unclear about how to user DEFMACRO-DRIVER etc. Could anybody help me? Thanks Marco -- Marco Antoniotti http://bioinformatics.nyu.edu NYU Courant Bioinformatics Group tel. +1 - 212 - 998 3488 715 Broadway 10th FL fax. +1 - 212 - 998 3484 New York, NY, 10003, U.S.A.

1 0

[iterate-devel] devel access?
by Hoehle, Joerg-Cyril 23 Feb '05

23 Feb '05

Hi, >Maybe it would be a good idea to also give a write access to >Jörg Höhle? I'm not at all against it. My experience with CLISP is that I prefer to get my stuff in *and then* let people possibly change it, instead of having somebody play the filter first in the line, breaking my stuff and causing me more work. However, I've some technical hurdles. I'm coming from within a corporate firewall, permitting FTP, HTTP and HTTPS and I think that's all. I manage to reach the CVS devel access to sourceforge (for CLISP) because they provide cvs-ssh (i.e. the connection looks to firewalls and proxies like a typical SSL one). I can't use anon-cvs AFAIK. I haven't yet used GNU Arch and already forgot what their webpages say about firewall interaction and limitations. What's your POV in this GNUArch/CVS/FW issue? Regards, Jörg Höhle

3 2

Re: [iterate-devel] iterate 1.3?
by Hoehle, Joerg-Cyril 23 Feb '05

23 Feb '05

Marco, >2) i'd like to make sure no one else has > other fixes lying around. Well, I have more stuff that I haven't yet sent to anyone. >1) antifuchs recently gave me write access to the iterate project on > common-lisp.net. i'd like to use my new power to create a new > release of iterate (there's been a lot of work since > 1.3pre1). anybody have problems with this? For the record, my code is like a fork on iterate-1.0.9. 'Antifuchs' 1.3pre1 is based on his augment-environment code, which I had somewhat promised to review, but never integrated into my code branch (for I was generally suspicious about macroexpansion-hooks to help macrolet). I haven't yet found the time to write about my recent review, although it's ready in my mind. The executive summary is that I demonstrate that it IMHO fixes too little for too many possibilities to break things, and I demonstrate that a *macroexpand-hook* cannot reliably expand macros or prevent the "expansion" of functions. That's general about *macroexpand-hook*. Then I also point at a concrete bug in the code. The only benefit of that code is that it makes nested in-hashtable iterations work. But I can provide examples with macrolet which will lead to incorrect code with 1.3pre1. My code I sent you in January/February was one with many bug fixes and few extensions, the last one I wanted to create before I would myself dig into and face the macrolet issue. My current code (not yet sent to anyone) (and which I consider not yet finished) does the following: + symbol-macrolet is accepted and does not cause expansion errors. The expansion must not generate iterate clauses, but that's a restriction people can live with, I believe. I added testcases involving with-slots and with-accessors. + macrolet is accepted, but even stronger restrictions apply. Iterate now signals a warning "don't know how to walk ..., so iterate clauses inside it will not be seen". That's as far as one can go with macrolet and still expect stability, robustness, and correctness. But it does not work with (iter outer in-hashtable (iter in-hashtable (in outer ...))) since that would require expansion within a macrolet body. I would like to release that (when it'll be ready), because + it is progress against any previous stable versions which simply rejected all occurrences of macrolet, so general code (not involving iterate) can now compile. + it is still stable. + releasing it would allow people to play with it and give feedback about these limitations. They will certainly be happy that Iterate now processes general code, e.g. containing CL:LOOP and loop-finish that result from expansion of arbitrary forms. Since then, I have noticed one more bug and added even more testcases (the lack of testcases is appaling, but I've steadily been increasing coverage). So here's what I suggest: + Please release what I sent you in January/February (not the 13th of December, which is quite "old" already :) + Give me time to write that review about *macroexpand-hook and augment-environment. Now that I said so much about it, I think I owe you all to write it up in detail. + Give me a little time to finish my current limited macrolet support + Then produce a new release with macrolet, yet another bugfix and more testcases that I've added and continue to add. Regards, Jorg Hohle.

1 0