Request for comments. Including grammatical and stylistic corrections.
-T.
Issue: WITH-READTABLE-ITERATOR
Forum: Common Lisp Document Repository (CDR)
Status: Draft.
References
* CLHS 2.1.1 Readtables
* X3J13 Issue #188
* `with-package-iterator' (macro)
* `with-hash-table-iterator' (macro)
Category: Addition.
Edit History: 02-Oct-2008 by Rittweiler (Draft)
Problem Description:
Even though the ANSI Common Lisp standard provides simple getters for
readtables (`get-macro-character', `get-dispatch-macro-character'), the
standard does not provide any means to efficiently get at all the macro
characters and dispatch macro characters defined in a readtable.
The omission of any iteration facility for readtables makes readtables
unneccessarily opaque, and keep users from writing libraries that try to deal
with readtables in a general way. For example, the author discovered that this
lack of an iteration form is the main obstacle to writing an otherwise
portable library that establishes an organizational namespace for readtables
akin to the namespace that is provided for packages.
Proposal (WITH-READTABLE-ITERATOR:ADD-GENERATOR)
Add a macro `with-readtable-iterator' that establishes a generator in its
scope; each invocation of this generator returns a macro character or a
dispatch macro character from the readtable the generator was established
for--along side some additional information.
A detailed specification of `with-readtable-iterator' can be found in the
appendix of this document.
Rationale
The proposed macro `with-readtable-iterator' represents a general iteration
facility for readtables that can be used to implement a variety of iteration
forms.
The proposal is closely modelled on the macros `with-hash-table-iterator' and
`with-package-iterator'. The semantics of `with-readtable-iterator' should
thus be intuitive to any experienced Common Lisp programmer.
Notes
The proposal does deliberately say nothing about the home-package of the
symbol `with-readtable-iterator'. However, implementors are encouraged to
export this symbol from their extensions package (often called "`EXT'") or
another appropriate package--unless a later CDR document specifies a more
explicit location.
Current Practice
No implementation the author is aware of provides a way to iterate through a
readtable.
The author implemented the proposal for SBCL (http://www.sbcl.org/) and sent
the relevant patches upstream; the patches are currently waiting to be
integrated into mainline. Ariel Badichi implemented the proposal for CLISP
(http://clisp.cons.org/) and is going to send his work upstream shortly.
Stephen Compall did an implementation for Clozure CL (http://ccl.clozure.com/)
which needs to be somewhat revised to fully conform to the proposal as
presented in this document.
Cost to implementators
The macro `with-readtable-iterator' should be straightforwardly
implementable. Extrapolating from actual experience, people--who were
previously not acquainted with the relevant code sections--were able to
implement it in a couple of hours.
Discussion
Ariel Badichi proposed coalescing a generator's fourth return value
(indicating if the returned character is a dispatch macro character) with its
first return value (indicating if the generator is exhausted.)
There is technically no reason that speaks against doing so; in fact, a
generator would return one value less this way--which may lead to positive
performance characteristics on register-anemic processor architectures.
Stephen Compall and the author opposed such a change mostly for idiomatic
reasons, as both `with-hash-table-iterator' and `with-package-iterator', the
generator-establishing macros specified by the ANSI standard, return a purely
boolean exhaustion flag as first value. In particular, `with-package-iterator'
does _not_ coalesce the accessibility type (third return value) with the
exhaustion flag (first return value.)
The author notes that allowing `:terminating', and `:non-terminating' as valid
MACRO-CHAR-TYPES was considered, but rejected for reasons of simplicity. It is
not apparent that there is a real necessity for supporting these out of the
box.
Acknowledgements
The author wants to specially credit Ariel Badichi and Stephen Compall which
in spirit of true hackerism promptly agreed to hack an early version of the
proposal into the implementations of their choice.
Appendix
-- Macro: with-readtable-iterator
(name readtable &rest macro-char-types) declaration* form*
=> results
Arguments and Values
....................
NAME - A symbol.
READTABLE - A form, evaluated once to produce a readtable.
MACRO-CHAR-TYPE - One of the symbols `:macro-char', or `:dispatch-macro-char'.
DECLARATION - A `declare' expression; not evaluated.
FORMS - An implicit progn.
RESULTS - The values of the FORMS.
Description
...........
Within the lexical scope of the body FORMS, the NAME is defined via `macrolet'
such that successive invocations of `(name)' will return the macro characters,
one by one, from the READTABLE. The order of the macro characters returned is
implementation-dependent.
An invocation of `(name)' returns the following five values:
1. A generalized boolean that is true if a macro character is returned.
2. A macro character that is defined in READTABLE.
3. A reader macro function of the macro character returned.
4. A generalized boolean that is true if the macro character is a dispatch
macro character.
5. An association list between the "sub-characters" of the dispatch macro
character and their reader macro functions.
After all macro characters have been returned by successive invocations of
`(name)', only one value is returned, namely `nil'.
It is unspecified what happens if any of the implicit interior state of an
iteration is returned outside the dynamic extent of the
`with-readtable-iterator' form such as by returning some closure over the
invocation form
In spirit of CLHS 3.6, consequences are undefined if READTABLE is modified
except for modification of the current macro character under traversal.
Exceptional Situations
......................
Signals an error of type `program-error' if a MACRO-CHAR-TYPE is supplied that
is not recognized by the implementation.
See Also
........
Traversal Rules and Side Effects (CLHS 3.6), `with-package-iterator'
