Request for comments. Including grammatical and stylistic corrections.
-T.
Issue: WITH-READTABLE-ITERATOR
Forum: Common Lisp Document Repository (CDR)
Status: Draft.
References
* CLHS 2.1.1 Readtables
* X3J13 Issue #188
* `with-package-iterator' (macro)
* `with-hash-table-iterator' (macro)
Category: Addition.
Edit History: 02-Oct-2008 by Rittweiler (Draft)
Problem Description:
Even though the ANSI Common Lisp standard provides simple getters for readtables (`get-macro-character', `get-dispatch-macro-character'), the standard does not provide any means to efficiently get at all the macro characters and dispatch macro characters defined in a readtable.
The omission of any iteration facility for readtables makes readtables unneccessarily opaque, and keep users from writing libraries that try to deal with readtables in a general way. For example, the author discovered that this lack of an iteration form is the main obstacle to writing an otherwise portable library that establishes an organizational namespace for readtables akin to the namespace that is provided for packages.
Proposal (WITH-READTABLE-ITERATOR:ADD-GENERATOR)
Add a macro `with-readtable-iterator' that establishes a generator in its scope; each invocation of this generator returns a macro character or a dispatch macro character from the readtable the generator was established for--along side some additional information.
A detailed specification of `with-readtable-iterator' can be found in the appendix of this document.
Rationale
The proposed macro `with-readtable-iterator' represents a general iteration facility for readtables that can be used to implement a variety of iteration forms.
The proposal is closely modelled on the macros `with-hash-table-iterator' and `with-package-iterator'. The semantics of `with-readtable-iterator' should thus be intuitive to any experienced Common Lisp programmer.
Notes
The proposal does deliberately say nothing about the home-package of the symbol `with-readtable-iterator'. However, implementors are encouraged to export this symbol from their extensions package (often called "`EXT'") or another appropriate package--unless a later CDR document specifies a more explicit location.
Current Practice
No implementation the author is aware of provides a way to iterate through a readtable.
The author implemented the proposal for SBCL (http://www.sbcl.org/), and sent the relevant patches upstream; the patches are currently waiting to be integrated into mainline. Ariel Badichi implemented the proposal for CLISP (http://clisp.cons.org/), and is going to send his work upstream shortly. Stephen Compall did an implementation for Clozure CL (http://ccl.clozure.com/) which needs to be somewhat revised to fully conform to the proposal as presented in this document.
Cost to implementators
The macro `with-readtable-iterator' should be straightforwardly implementable. Extrapolating from actual experience, people--who were previously not acquainted with the relevant code sections--were able to implement it in a couple of hours.
Discussion
Ariel Badichi proposed coalescing a generator's fourth return value (indicating if the returned character is a dispatch macro character) with its first return value (indicating if the generator is exhausted.)
There is technically no reason that speaks against doing so; in fact, a generator would return one value less this way--which may lead to positive performance characteristics on register-anemic processor architectures.
Stephen Compall and the author opposed such a change mostly for idiomatic reasons, as both `with-hash-table-iterator' and `with-package-iterator', the generator-establishing macros specified by the ANSI standard, return a purely boolean exhaustion flag as first value. In particular, `with-package-iterator' does _not_ coalesce the accessibility type (third return value) with the exhaustion flag (first return value.)
The author notes that allowing `:terminating', and `:non-terminating' as valid MACRO-CHAR-TYPES was considered, but rejected for reasons of simplicity. It is not apparent that there is a real necessity for supporting these out of the box.
Acknowledgements
The author wants to specially credit Ariel Badichi and Stephen Compall which in spirit of true hackerism promptly agreed to hack an early version of the proposal into the implementations of their choice.
Appendix
-- Macro: with-readtable-iterator (name readtable &rest macro-char-types) declaration* form* => results
Arguments and Values ....................
NAME - A symbol.
READTABLE - A form, evaluated once to produce a readtable.
MACRO-CHAR-TYPE - One of the symbols `:macro-char', or `:dispatch-macro-char'.
DECLARATION - A `declare' expression; not evaluated.
FORMS - An implicit progn.
RESULTS - The values of the FORMS.
Description ...........
Within the lexical scope of the body FORMS, the NAME is defined via `macrolet' such that successive invocations of `(name)' will return the macro characters, one by one, from the READTABLE. The order of the macro characters returned is implementation-dependent.
An invocation of `(name)' returns the following five values:
1. A generalized boolean that is true if a macro character is returned.
2. A macro character that is defined in READTABLE.
3. A reader macro function of the macro character returned.
4. A generalized boolean that is true if the macro character is a dispatch macro character.
5. An association list between the "sub-characters" of the dispatch macro character and their reader macro functions.
After all macro characters have been returned by successive invocations of `(name)', only one value is returned, namely `nil'.
It is unspecified what happens if any of the implicit interior state of an iteration is returned outside the dynamic extent of the `with-readtable-iterator' form such as by returning some closure over the invocation form
In spirit of CLHS 3.6, consequences are undefined if READTABLE is modified except for modification of the current macro character under traversal.
Exceptional Situations ......................
Signals an error of type `program-error' if a MACRO-CHAR-TYPE is supplied that is not recognized by the implementation.
See Also ........
Traversal Rules and Side Effects (CLHS 3.6), `with-package-iterator'