- iterate-devel - mailman3.common-lisp.net

Re: [iterate-devel]/[series-users] collect-tail ?
by Hoehle, Joerg-Cyril 19 Dec '05

19 Dec '05

Chris, since you're asking similar questions in series-users and iterate-devel, I'm starting to wonder whether you're going to write a paper on comparisons of some looping frameworks? :-) >how do I go about collecting the last N elements of anything? > (iter (for line :in-file my-file :using #'read-line) > (collect-tail line :how-many 10)) If 10 is known at macro-expansion time, you can always write a macro that expands to a series of 10 uses of PREVIOUS in Iterate, or 10 consecutive latches in Series. I.e., IMHO there are always two ways, either with series of iterate: a) Either take a code generation approach and create code that latches (= delay by one) N times, b) Or take a functional approach, implement a ring datastructure and use that. The former looks like a better use of Series: it could be a series of the 10 last values (as a list, or 10 series of values farther and farther behind). In/After the last iteration, you'll have the 10 last values. As to which approach is better... Probably you wouldn't want to generate code for the last 100 values. BTW, you'll have to defined what happens when less than 10 iterations were taken. Straight-forward code is likely to return (val1 val2 nil nil ...6xnil), whereas one could wish for (val1 val2) with only 2 iterations. Regards, Jorg Hohle.

2 1

[iterate-devel] collect-tail ?
by Chris Dean 15 Dec '05

15 Dec '05

How would I go about reading the last 10 lines in a files? Or more generally, how do I go about collecting the last N elements of anything? I'm looking for advice on the "right way" to do this using iterate. On way would be to write collect-tail that just kept the last N elements: (iter (for line :in-file my-file :using #'read-line) (collect-tail line :how-many 10)) Is this a good way? Regards, Chris Dean

1 0

[iterate-devel] Between-each, FORMAT's ~^ and code-walking
by Hoehle, Joerg-Cyril 09 Dec '05

09 Dec '05

Mészáros Levente wrote: >> Would you like to show more elaborate loop examples? >Let's say one has a list and one wants to transform it into a >string by transforming each element to a string and putting a >comma between them. >(concatenate-as-string > (iter (for e in l) > (except-first-time (collect ", ")) > (collect (as-string e)))) I recently came across that article about Lispy FORMAT replacements and started wondering how to nicely implement FORMAT's "~^" idiom. http://www.cs.yale.edu/homes/dvm/format-stinks.html BTW, I haven't seen the code for McDermott's out macro, but I expect it to be subject to code-walking problems similar to Iterate, series and other packages that depend on traversing code. Why? consider one example from the above article: (out "#<Foo " (Foo-label f1) (:q ((eq (Foo-status f1) ':abnormal) "?")) (:e (dolist (x (Foo-contents f1)) (:o " " x))) ">") Do you see the :o keyword inside the dolist? I think keywords are constants and cannot be locally bound (using MACROLET). Therefore, I believe the OUT macro needs to code-walk its body to track down all uses of :o and insert the replacememt code. Had a non-constant symbol been used therein, then MACROLET could have been used and the implementation of the macro out would (presumably) be clean. In any case, ~^ is quite different from *-first-time. It's more like an early test of loop termination. The macro out does not propose an idiom for that. The examples just repeat the termination test, as in (defun print-xapping ...) ... (do ((d ... (cdr d))) ... (:q ((not (null (cdr d))) " "))))) Repeating the test makes it unable to use the much more readable DOLIST instead of DO, and therefore must also use (CAR d) to access the elements. Reading the full paper (ytools) instead of the short FORMAT critique, it appears to me that OUT is no silver bullet either. It has its own set of controls: :a, :e, :t, :i>, :i<, :f, :v, :pp-block etc., so it's still complex to grok. I found the difference more appealing between e.g. the sexp-based regular expression notation (IIRC there's a Scheme SRFI about it) that the error-prone flat string syntax. BTW, McDermott's diver&snorkeler macro design pattern is interesting. Regards, Jörg Höhle

2 1

Re: [iterate-devel] Between-each
by Hoehle, Joerg-Cyril 02 Dec '05

02 Dec '05

Hi, Mészáros Levente wrote: >I may be wrong about this, but I could not find support for a >quite commonly used loop pattern, namely when you want >something to be evaluated between each loop cycles. Please accept my apology for responding that late. I felt a little uncomfortable (see below) and did not quite know how to reply. >(def-special-clause BETWEEN-EACH (between) > (return-code :body (list (if-1st-time nil (list (walk-expr between)))))) Correct code (like implementation of IF-FIRST-TIME). >Maybe I'm wrong, could somebody tell how to do this with >iterate? If there's no support for this perhaps you could put >something like this into the main branch. You could use a macro-substitution and expand to the exported and documented IF-FIRST-TIME and DEFMACRO-CLAUSE, instead of using package internals like return-code. Your implementation shows that there's the same relationship between IF-FIRST-TIME and your BETWEEN-EACH like there's between WHEN and UNLESS. So why not add it, even it would be redundant to purists? However, given that analogy, I find the given name quite misleading, as it's very different from AFTER-EACH. >And it works like: >(iter (for i from 1 to 3) (between-each (collect (+ i 100))) > (collect i)) >(1 102 2 103 3) Hmm, I feel uncomfortable with the name, given the functionality you show. "Between" is something I'd expect to happen after one iteration, while you use it within the body. I.e., my wording shows I'm thinking in terms of AFTER-EACH. Well, there's AFTER-EACH, which is subject to code movement, but wouldn't it work for you? Another question is: how does this extension behave when used outside the top-level of the loop. Does the given name adequately reflect the functionality? E.g. (iter (for i...) (if (oddp i) (between-each (collect i)))) What would a reader expect as value from such a form? What about BUT-FIRST[-TIME] as a name? Would you like to show more elaborate loop examples? Regards, Jörg Höhle.

1 0

Re: [iterate-devel] fishy iterate use of delete...
by Hoehle, Joerg-Cyril 02 Nov '05

02 Nov '05

Faré wrote: >Indeed, (delete ...) has side-effects and doesn't preserve its second >argument. Although I can hardly imagine any implementation where DELETE on lists would operate in ways that break the code (except for an ANSI-compliance stress tester mode), it's not correct use of DELETE by the CLHS. Thus, a minimally disruptive patch would be to replace >(delete entry index) with (setf index (delete entry index)) twice, Funny. It seems the code has an extra element in the list precisely to avoid the well-known "delete yields empty list not reflected in variable" Lisp FAQ issue, but CLHS precisely disables such optimization when using DELETE (instead of a self-written naive implementation). >Someone who understands the code >better audit what is being done by the considered functions. > and (delete nil temps) with (remove nil temps). Assuming you're talking about make-mv-dsetqs: (let ((mv-setq `(multiple-value-setq ,vars ,value)) (temp-vars (delete nil temps))) Great, that's not just being picky about standard compliance, you discovered a real bug in the code! temps is still used afterwards and DELETE destroys it. REMOVE MUST be used instead. Here's a testcase I came up with: (let (x y z) (dsetq (values nil x (y z)) (values 1 'a '(b c))) (list x y z)) must yield (a b c), not (a nil nil). Regards, Jörg Höhle.

1 0

[iterate-devel] fishy iterate use of delete...
by Faré 02 Nov '05

02 Nov '05

Dear Iterate developers, while I was compiling cl-pdf, sbcl detected some invalid uses of delete in iterate.lisp. I checked that these errors were there in the iterate 1.4 release available from asdf-install. Indeed, (delete ...) has side-effects and doesn't preserve its second argument. Thus, a minimally disruptive patch would be to replace (delete entry index) with (setf index (delete entry index)) twice, and (delete nil temps) with (remove nil temps). Someone who understands the code better audit what is being done by the considered functions. Patch attached. Happily these errors didn't seem to affect the behaviour of cl-pdf. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] You may easily play a joke on a man who likes to argue -- agree with him. -- Ed Howe

2 1

[iterate-devel] Between-each
by Mészáros Levente 25 Oct '05

25 Oct '05

Hi, I may be wrong about this, but I could not find support for a quite commonly used loop pattern, namely when you want something to be evaluated between each loop cycles. It means if the loop runs N times it will be evaluated N - 1. I did a patch like: (def-special-clause BETWEEN-EACH (between) (return-code :body (list (if-1st-time nil (list (walk-expr between)))))) And it works like: (iter (for i from 1 to 3) (between-each (collect (+ i 100))) (collect i)) (1 102 2 103 3) Maybe I'm wrong, could somebody tell how to do this with iterate? If there's no support for this perhaps you could put something like this into the main branch. Please send it to my direct mail address too, because I'm not on the list. Levy

1 0

[iterate-devel] final version of "conservative" Iterate: 1.4
by Hoehle, Joerg-Cyril 23 Jun '05

23 Jun '05

Hi, I've sent a new version of Iterate to Marco, Marc, Kevin, Peter and a few others, so it should appear in the usual places soon. Changes are too numerous to mention (and it depends on which version you're currently using:) and still extremely conservative. I'll have to supply the updated iter-man.pdf a little later, because my current Ubuntu laptop generates much uglier fonts therein than the old Suse one (maybe the default printer which is ljfour is the reason??). o supports nested in-hashtable and in-package o adjoin :at :start did not honour :test o fixed in-file/in-stream destructuring o do-next forgot about finally etc. clauses o works as case-sensitive package in CLISP (don't know about Allegro) o various optimizations, less bugs, more tests o tested with clisp, cmucl, sbcl, gclcvs This is what I consider the last version of 1.4, called "conservative". 1.5 will do some possibly incompatible changes to allow more optimizations (see recent e-mail of mine about my version plan). Regards, Jörg Höhle.

1 0

Re: [iterate-devel] Version plan for Iterate 1.4 and 1.5
by Hoehle, Joerg-Cyril 17 Jun '05

17 Jun '05

Marc Battyani wrote: >>code-movement [possible changes] >Those parts (prologue/epilog/var bindings) are under- (and >badly) specified >in LOOP and implementations differ on this. So let's keep it >in a sane way. I find it underspecified in Iterate as well. I want to change that state :-) E.g. Andreas' with-*-iterator based in-hashtable and in-package has very different initialization behaviour from the old ones (see below). It occured to me that there's another issue: iterate-pg may not work as expected, for similar reasons. (iter (with foo = connection) (with x on-relation foo connection bar)) #) ;won't work Reason: WITH-PG-TRANSACTION is move outside the whole Iterate form, thus foo and bar get evaluated before any other thing in Iterate(!) My current belief is that the following order would be a better match: (let (...) (wrappers: unwind-protect and with-pg-transaction, in order of apperance (block (tagbody #)))) Currently, it's (wrappers: with-package-iterator, with-hashtable-iterator, with-pg-transaction (let # (unwind-protect (block (tagbody #))))) Therefore (iter (with table = (foo)) (for (k v) in-hashtable table)) used to work with the old maphash based Iterate, but fails since Andreas' changes in 2004. I think this is also a result of underspecification of Iterate, together with a quite complex package and lots of feature interactions (initialization time, generators, declarations, top-level only clauses, support for PREVIOUS etc.) As a result, simple examples (e.g. the testsuite) work, but subtle forms fail. I'm going to change that as well (1.5). At the present, one can think of two forms of body wrappers: a) around the whole body (e.g. current with-*-iterator) b) around BLOCK, but inside LET (e.g. unwind-protect) I think I'm going to provide for only variant b, as shown above. Subtle interactions between unwind-protect clauses can still happen. I think that's a sign of limitations of Iterates design. All in all, packages needing a code-walker are not that great (they appear like a can of worms, e.g. Screamer was well, with some forms that work as expected and others that fail surprisingly (cf. primordial in Screamer). There's quite some truth when Bruno Haible expresses that CL somehow fails to be its own extension language. LOOP's weakness is also its strength: not needing a code walker, being limited to top-level forms, having a grammar so its transformation does not require macroexpand and remains syntactic. Regards, Jörg Höhle

1 0

[iterate-devel] Iterate in-hashtable expansion, with-hashtable-iterator vs. mapha sh
by Hoehle, Joerg-Cyril 15 Jun '05

15 Jun '05

Hi, first, some good news: I think I found an easy way to make Iterate support nested in-hashtable, together with with-hashtable-iterator: Just map with-hashtable-iterator to walk-cddr-with-declarations in *special-form-alist* That should basically work good enough for a start, but it does not walk the hashtable parameter. Next: a possible change -- what do you think? Some packages like SCREAMER have explicitly chosen to support Iterate instead of loop because they walk code and suffer from similar limitations about macrolet as Iterate does. In the past, Iterate never expanded to forms that Screamer was unable to analyse. Starting with Andreas Fuchs' change, he made IN-HASHTABLE expand to with-package-iterator, which is based on MACROLET. Similarly for in-package. As a result, Screamer would error out should it be updated to use one of ours iterate.lisp. Therefore, I'm thinking about introducing a choice, either as a *settable-variable*, or as new iteration keywords, for how to Iterate through hashtables (and packages). I first thought about in-hashtable: use with-hashtable-iterator on-hashtable: use maphash to build up list upfront. Then I realized that it should rather be the other way round, or somebody would have to modify the Screamer examples, and all other unknown packages built using Screamer (and possibly Series, I don't know). An alternative may be to get Screamer support macrolet, but that's IMHO even farther away than having Iterate do that. So what do you prefer? a) in-hashtable -> maphash, on-hashtable -> with-*-iterator and same for packages "reintroduce old behaviour" b) on-hashtable -> maphash, in-hashtable -> with-*-iterator and same for packages "affirm changed behaviour, need to change random packages" c) defvar iterate::*use-with-iterators* Screamer could set such a variable, or at least the examples should do so (since only the examples depend on Iterate, not Screamer itself) I'm biased towards b). Most users will benefit from the iterator, and I'd need to convince the Screamer authors anyway to package a more recent version of Iterate with it, so the examples could be updated at that time, or to at least have the debian cl-screamer package not include but depend on cl-iterate (and similarly for cl-pdf etc.). Still, the users would have to change their Screamer code. (and no, I haven't yet tested Screamer with my version of Iterate, this is just code review and thinking loud, yet I expect no problems beyond that). BTW, having a maphash-based iterator allows to modify any hashtable item while iterating it, instead of just the current one (forbidden by ANSI-CL and many other language specifications). That can come handy sometime. While we are at it, there's a difference between the old in-package and the new (since Andreas Fuchs in 2004): the new (with-package-iterator based) one does not evaluate the :external-only argument any more -- How I hate it when documentations does not precisely state what is evaluated and what not! it always makes simple examples work and complex ones fail. With-package-iterator does not provide for evaluation of this, while the old in-hashtable dispatched to either do-symbols or do-external-symbols. So far, I plan to stick to that change, or does anybody wave hands? Regards, Jörg Höhle.

1 0