Update of /project/climacs/cvsroot/climacs/Doc
In directory common-lisp.net:/tmp/cvs-serv32318
Modified Files:
climacs-internals.texi
Log Message:
Common Lisp syntax section.
Date: Sat Dec 25 14:23:01 2004
Author: rstrandh
Index: climacs/Doc/climacs-internals.texi
diff -u climacs/Doc/climacs-internals.texi:1.1 climacs/Doc/climacs-internals.texi:1.2
--- climacs/Doc/climacs-internals.texi:1.1 Sat Dec 25 13:05:17 2004
+++ climacs/Doc/climacs-internals.texi Sat Dec 25 14:22:59 2004
@@ -445,6 +445,8 @@
@chapter Redisplay and the syntax protocol
+@section General
+
A buffer can be on display in several panes. The redisplay
algorithm is invoked on each such pane. Each pane is associated
with a distinguished mark called the `point' of the pane. The point
@@ -511,6 +513,781 @@
parser needs to use the low-mark and high-mark to determine which
parts of the buffer have changed, and recompute parsing information as
necessary.
+
+@section Common Lisp syntax
+
+Technically, comments and such are not expressions, but it is OK for
+our purposes to consider that they are.
+
+@multitable @columnfractions .2 .1 .5
+@item program @tab -> @tab expr*
+@item expr @tab -> @tab quoted-expression
+@item expr @tab -> @tab comment
+@item expr @tab -> @tab string
+@item expr @tab -> @tab backquote-expression
+@item expr @tab -> @tab unquote-expression
+@item expr @tab -> @tab list-expression
+@item expr @tab -> @tab label-ref
+@item expr @tab -> @tab function-expression
+@item expr @tab -> @tab vector-expression
+@item expr @tab -> @tab array-expression
+@item expr @tab -> @tab bitvector-expression
+@item expr @tab -> @tab labeled-expression
+@item expr @tab -> @tab character
+@item expr @tab -> @tab balanced-comment
+@item expr @tab -> @tab read-time-conditional-+
+@item expr @tab -> @tab read-time-conditional--
+@item expr @tab -> @tab read-time-evaluation
+@item expr @tab -> @tab binary-expression
+@item expr @tab -> @tab complex-expression
+@item expr @tab -> @tab octal-expression
+@item expr @tab -> @tab pathname-expression
+@item expr @tab -> @tab radix-n-expression
+@item expr @tab -> @tab hex-expression
+@item expr @tab -> @tab token
+@end multitable
+
+@multitable @columnfractions .3 .3 .3
+@headitem Expression @tab Syntax @tab handle/read
+@item quoted-expression @tab 'expr @tab handle
+@item comment @tab ; chars <newline> @tab handle
+@item string @tab " chars " @tab read
+@item backquote-expression @tab ` expr @tab handle
+@item unquote-expression @tab , expr @tab handle
+@item list-expression @tab ( expr* ) @tab handle
+@item label-ref @tab #n# @tab handle
+@item function-expression @tab #' expr @tab handle
+@item vector-expression @tab #( expr* ) @tab handle
+@item array-expression @tab #nA(...) @tab handle
+@item bitvector-expression @tab #* bits @tab read
+@item labeled-expression @tab #n= @tab handle
+@item character @tab #\.. @tab read
+@item balanced-comment @tab #| .. |# @tab handle
+@item read-time-conditional-+ @tab #+test expr @tab handle
+@item read-time-conditional-- @tab #-test expr @tab handle
+@item read-time-evaluation @tab #.expr @tab handle
+@item binary-expression @tab #b... @tab read
+@item complex-expression @tab #c... @tab read
+@item octal-expression @tab #o... @tab read
+@item pathname-expression @tab #p... @tab read
+@item radix-n-expression @tab #nr @tab read
+@item hex-expression @tab #x @tab read
+@end multitable
+
+@deftp {class} stack-entry
+
+A stack entry corresponds to a syntactic category and contains a
+start offset and an end offset. The end offset is initialized to
+nil, meaning we do not know where it ends.
+@end deftp
+
+@deftp {initarg} :start-offset
+@end deftp
+
+@deftp {class} error-entry
+
+This is a subclass of stack-entry. It is used whenever some invalid
+input syntax is encountered.
+@end deftp
+
+@deftp {class} terminal-entry
+
+This is a subclass of stack-entry. It is used for tokens (numbers,
+symbols), but also for macro characters that start more complex
+expressions.
+@end deftp
+
+@deftp {class} number-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+numbers.
+@end deftp
+
+@deftp {class} symbol-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+symbols.
+@end deftp
+
+@deftp {class} character-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+characters.
+@end deftp
+
+@deftp {class} string-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+character strings
+@end deftp
+
+@deftp {class} quote-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+quote inidicators.
+@end deftp
+
+@deftp {class} backquote-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+backquote indicators.
+@end deftp
+
+@deftp {class} unquote-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+unquote indicators.
+@end deftp
+
+@deftp {class} comment-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+single-line comment indicators.
+@end deftp
+
+@deftp {class} list-start-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+list start indicators.
+@end deftp
+
+@deftp {class} list-end-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+list end indicators.
+@end deftp
+
+@deftp {class} label-ref-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+label reference indicators.
+@end deftp
+
+@deftp {class} label-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+label indicators.
+@end deftp
+
+@deftp {class} function-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+function indicators.
+@end deftp
+
+@deftp {class} balanced-comment-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+balanced comment entry indicators.
+@end deftp
+
+@deftp {class} read-time-conditional-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+read-time conditional indicators.
+@end deftp
+
+@deftp {class} vector-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+vector indicators.
+@end deftp
+
+@deftp {class} array-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+array indicators.
+@end deftp
+
+@deftp {class} bitvector-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+bit vector indicators.
+@end deftp
+
+@deftp {class} read-time-evaluation-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+read-time evaluation indicators.
+@end deftp
+
+@deftp {class} complex-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+complex indicators.
+@end deftp
+
+@deftp {class} octal-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+octal rational indicators.
+@end deftp
+
+@deftp {class} hex-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+hex rational indicators.
+@end deftp
+
+@deftp {class} radix-n-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+radix-n rational indicators.
+@end deftp
+
+@deftp {class} pathname-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+pathname indicators.
+@end deftp
+
+@deftp {class} binary-entry
+
+A subclass of terminal-entry corresponding to the syntactic class of
+binary rational indicators.
+@end deftp
+
+@deftp {class} unknown-entry
+
+A subclass of terminal-entry corresponding to unknown (user-defined)
+syntactic classes.
+@end deftp
+
+@deftp {class} nonterminal-entry
+
+This is a subclass of stack-entry. A nonterminal entry maintains an
+expression count, which is initialized to 0. It corresponds to the
+number of subexpressions of this expression that have been detected.
+A nonterminal entry also maintains a backquote-depth which is
+initially 0, is incremented by 1 for each nested backquote, and is
+decremented by 1 for each comma.
+@end deftp
+
+@deffn {generic function} expression-count nonterminal-entry
+
+Return the expression count of this entry.
+@end deffn
+
+@deffn {generic function} {(setf expression-count)} count nonterminal-entry
+
+Set the expression count of this entry.
+@end deffn
+
+@deffn {generic function} backquote-depth nonterminal-entry
+
+Return the backquote-depth of this entry.
+@end deffn
+
+@deffn {generic function} {(setf backquote-depth)} depth nonterminal-entry
+
+Set the backquote-depth of this entry.
+@end deffn
+
+@deftp {class} program-entry
+
+A subclass of nonterminal-entry corresponding to the entire buffer
+contents.
+@end deftp
+
+@deftp {class} quoted-expression-entry
+
+A subclass of nonterminal-entry corresponding to a quoted
+expression.
+@end deftp
+
+@deftp {class} unquoted-expression-entry
+
+A subclass of nonterminal-entry corresponding to an unquoted
+expression.
+@end deftp
+
+@deftp {class} list-expression-entry
+
+A subclass of nonterminal-entry corresponding to a list expression.
+@end deftp
+
+@deftp {class} function-expression-entry
+
+A subclass of nonterminal-entry corresponding to a function
+expression.
+@end deftp
+
+@deftp {class} vector-expression-entry
+
+A subclass of nonterminal-entry corresponding to a vector expression.
+@end deftp
+
+@deftp {class} labeled-expression-entry
+
+A subclass of nonterminal-entry corresponding to a labeled expression.
+@end deftp
+
+@deftp {class} array-expression-entry
+
+A subclass of nonterminal-entry corresponding to a list expression.
+@end deftp
+
+@deftp {class} array-dimension-entry
+
+A subclass of nonterminal-entry corresponding to an array dimension.
+@end deftp
+
+@deftp {class} read-time-feature-entry
+
+A subclass of nonterminal-entry corresponding to a read-time
+feature.
+@end deftp
+
+@deftp {class} read-time-conditional-expression-entry
+
+A subclass of nonterminal-entry corresponding to a read-time
+conditional expression.
+@end deftp
+
+@deftp {class} read-time-evaluation-expression-entry
+
+A subclass of nonterminal-entry corresponding to a read-time
+evaluation expression.
+@end deftp
+
+@deftp {class} complex-expression-entry
+
+A subclass of nonterminal-entry corresponding to a complex
+expression.
+@end deftp
+
+@deftp {class} octal-expression-entry
+
+A subclass of nonterminal-entry corresponding to an octal expression.
+@end deftp
+
+@deftp {class} hex-expression-entry
+
+A subclass of nonterminal-entry corresponding to a hexadecimal
+expression.
+@end deftp
+
+@deftp {class} radix-n-expression-entry
+
+A subclass of nonterminal-entry corresponding to a radix-n
+expression.
+@end deftp
+
+@deftp {class} pathname-expression-entry
+
+A subclass of nonterminal-entry corresponding to a pathname
+expression.
+@end deftp
+
+@deftp {class} binary-expression-entry
+
+A subclass of nonterminal-entry corresponding to binary expression.
+@end deftp
+
+@deftp {class} buffer-stream
+
+A stream corresponding to a text buffer. It contains a current parse
+stack which is a list of stack entries.
+@end deftp
+
+@deftp {initarg} :buffer
+
+The buffer underlying the stream.
+@end deftp
+
+@deffn {generic function} current-offset buffer-stream
+
+Return the current offset of the buffer stream.
+@end deffn
+
+@deffn {generic function} {(setf current-offset)} offset buffer-stream
+
+set the current offset of the buffer stream.
+@end deffn
+
+@deffn {generic function} max-offset buffer-stream
+
+Return the maximum offset of the buffer stream, beyond which no
+parsing is required.
+@end deffn
+
+@deffn {generic function} {(setf max-offset)} offset buffer-stream
+
+set the maximum offset of the buffer stream.
+@end deffn
+
+@deffn {method} stream-read-char buffer-stream
+
+Return the character at the current offset (or :eof if the offset is
+equal to the size of the buffer)
+@end deffn
+
+@deffn {generic function} parse-stack buffer-stream
+
+Return the current parse stack of the buffer stream. The parse stack
+contains either all nonterminal, or all but the top element
+nonterminal. The list of entries is initially a list of a single
+entry corresponding to the syntactic category `program-entry' with a
+start offset of 0 and an expression count of 0.
+@end deffn
+
+@deffn {generic function} {(setf parse-stack)} stack buffer-stream
+
+Set the current parse stack of the buffer.
+@end deffn
+
+@deffn {generic function} analysis buffer-stream
+
+An adjustable vector each element of which is a parse stack with top
+element being terminal. The vector is initially empty.
+@end deffn
+
+@deffn {generic function} invalidate-parse buffer-stream offset
+
+Called by client code to indicate that the buffer has been altered
+starting at the offset indicated. The analysis vector of the buffer
+stream is shortened to the last entry ending before or at the
+offset. The parse stack is set to the tail of the stack at the last
+entry of the analysis vector. The current offset is set to the end
+offset of the last entry of the analysis vector.
+@end deffn
+
+@deffn {generic function} advance-parse buffer-stream offset
+
+Inform the parser that it should construct a valid analysis vector up
+to the offset indicated. The redisplay module will call this
+function with the highest offset of all the windows on display before
+calling display-region for each window. The algorithm is as follows:
+
+ Set the max offset of the buffer stream to the offset passed as
+ argument.
+ Loop until the current offset is greater than or equal to max
+ offset (i.e., EOF on the buffer stream is reached):
+ Remember the current offset
+ Read a character
+ If it is a whitespace,
+ do nothing
+ Else if it is a single-dispatch-macro character
+ Call parse-macro-character with the buffer stream, the value of
+ a call to get-macro-character of the character, the
+ top-element of the current parse stack, and the remembered
+ offset.
+ Else
+ Unread the character and call `read' on the stream.
+ Add an entry to the end of the analysis vector containing as
+ start offset the remembered offset, as end offset the current
+ offset after the call to read, and a stack containing the
+ current parse stack with an additional entry on top
+ corresponding to the item read (number, symbol, etc).
+ Increment the expression count of the top of the stack.
+ Perform a count check (see below).
+@end deffn
+
+@deffn {Generic Function} parse-macro-character buffer-stream function entry o
+
+This generic function is called with a buffer stream, with a function
+that is returned by a call to get-macro-character, the top-entry
+of the current parse stack, and a remembered offset.
+@end deffn
+
+It has a number of methods, each one specialized (using an eql
+specializer) on a function returned by a call to get-macro-character
+in standard syntax. We indicate here the character itself, but it is
+the associated function that is actually used.
+
+@table @samp
+@item ) nonterminal-entry
+@itemize @bullet
+@item
+Add an entry at the end of the analysis vector with the current
+stack augmented with an entry for `error-entry' with a start offset
+of the remembered offset and an end offset of the current offset.
+@end itemize
+@item ) vector-expression-entry
+@itemx ) list-expression-entry
+@itemx ) array-expression-entry
+@itemize @bullet
+@item
+Set the end offset of the entry on top of the stack to the current
+offset.
+@item
+Pop the current stack.
+@item
+Increment the expression count of the top of the stack.
+@item
+Perform a count check (see below)
+@item
+Add an entry at the end of the analysis vector with the current
+stack augmented with an entry for `list-end-entry' with a start
+offset of the remembered offset, and an end offset of the current
+offset
+@end itemize
+
+@item ) nonterminal-entry
+@itemize @bullet
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `error-entry' with a start offset
+ of the remembered offset and an end offset of the current offset.
+@end itemize
+
+@item ) vector-expression-entry
+@itemx ) list-expression-entry
+@itemx ) array-expression-entry
+@itemize @bullet
+@item
+ Set the end offset of the entry on top of the stack to the current
+ offset.
+@item
+ Pop the current stack.
+@item
+ Increment the expression count of the top of the stack.
+@item
+ Perform a count check (see below).
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `list-end-entry' with a start
+ offset of the remembered offset, and an end offset of the current
+ offset.
+@end itemize
+
+@item ; nonterminal-entry
+@itemize @bullet
+@item
+ Read characters until a newline has been read.
+@item
+ Add an entry to the end of the analysis vector with the current
+ stack augmented with an entry for `comment-entry' with a start
+ offset of the remembered offset and the end offset of the current
+ offset.
+@end itemize
+
+@item ' nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for quoted-expression-entry on the stack with a
+ backquote-depth of one plus that of the current top of the stack.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `quote-entry' with a
+ start offset of the remembered offset and an end offset of
+ the current offset.
+@end itemize
+
+@item ` nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for quoted-expression-entry on the stack with a
+ backquote-depth of that of the current top of the stack plus one.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `backquote-entry' with a
+ start offset of the remembered offset and an end offset of
+ the current offset.
+@end itemize
+
+@item , nonterminal-entry
+@itemize @bullet
+@item
+ If the backquote-depth of the top of the stack is greater than 0
+@itemize @bullet
+@item
+ Push an entry for unquoted-expression-entry on the stack with a
+ backquote-depth of that of the current top of the stack minus
+ one.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `unquote-entry' with a
+ start offset of the remembered offset and an end offset of
+ the current offset.
+@end itemize
+@item
+ Else
+@itemize @bullet
+@item
+ Push an entry for unquoted-expression-entry on the stack with a
+ backquote-depth of 0.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `error-entry' with a start
+ offset of the remembered offset and an end offset of the current
+ offset.
+@end itemize
+@end itemize
+
+@item ( nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for list-expression-entry on the stack with a
+ backquote-depth of that of the current top of the stack.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `list-start-entry' with a
+ start offset of the remembered offset and an end offset of
+ the current offset.
+@end itemize
+
+@item " nonterminal-entry
+@itemize @bullet
+@item
+ Call the function associated with the character.
+ Increment the expression count of the top of the stack.
+@item
+ Perform a count check (see below).
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `string-entry' with a start
+ offset of the remembered offset and an end offset of the current
+ offset.
+@end itemize
+
+@item # nonterminal-entry
+@itemize @bullet
+@item
+ Remember the first char.
+@item
+ Read characters as long as they are digits, and form a count (a
+ number or nil if there are no digits).
+@item
+ If the first non-digit character is a dispatch macro character
+@itemize @bullet
+@item
+ Call parse-dispatch-macro-character with the buffer stream, the
+ result of a call to get-dispatch-macro-character, the entry, and
+ the remembered offset
+@end itemize
+@item
+ Else
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `error-entry' with a start
+ offset of the remembered offset and an end offset of the current
+ offset.
+@end itemize
+
+@item any nonterminal-entry
+@itemize @bullet
+@item
+ Call the function associated with the character.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `unknown-entry' with a start
+ offset of the remembered offset and an end offset of the
+ current offset.
+@end itemize
+@end table
+
+@deffn {Generic Function} parse-dispatch-macro-character buffer-stream function entry o
+@end deffn
+
+@table @samp
+@item # nonterminal-entry
+@itemize @bullet
+@item
+ Increment the expression count of the top of the stack.
+@item
+ Perform a count check (see below).
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `label-ref-entry' with a
+ start offset of the remembered offset and an end offset of
+ the current offset.
+@end itemize
+
+@item ' nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for function-expression-entry on the stack with a
+ backquote-depth of that of the current top of the stack.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with the entry for function-entry with a start
+ offset of the remembered offset and the end offset of the current
+ offset.
+@end itemize
+
+@item ( nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for vector-expression-entry on the stack with a
+ backquote-depth of that of the current top of the stack.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with the entry for vector-entry with a start offset
+ of the remembered offset and the end offset of the current offset.
+@end itemize
+
+@item A nonterminal-entry
+ [to be filled in]
+
+@item = nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for labeled-expression-entry on the stack with a
+ backquote-depth of that of the current top of the stack.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with the entry for label-entry with a start offset
+ of the remembered offset and the end offset of the current offset.
+@end itemize
+
+@item | nonterminal-entry
+@itemize @bullet
+@item
+ Read characters until a |# pair has been seen or EOF is reached.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with the entry for balanced-comment-entry with a
+ start offset of the remembered offset and the end offset of the
+ current offset.
+@end itemize
+
+@item + nonterminal-entry
+@itemx - nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for read-time-conditional-expression-entry on the
+ stack with a backquote-depth of that of the current top of the
+ stack.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with the entry for read-time-conditional-entry with
+ a start offset of the remembered offset and the end offset of the
+ current offset.
+@end itemize
+
+@item . nonterminal-entry
+@itemize @bullet
+@item
+ Push an entry for read-time-evaluation-expression-entry on the
+ stack with a backquote-depth of that of the current top of the
+ stack.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with the entry for read-time-evaluation-entry with
+ a start offset of the remembered offset and the end offset of the
+ current offset.
+@end itemize
+
+@item any nonterminal-entry
+@itemize @bullet
+@item
+ Call the function associated with the character.
+@item
+ Add an entry at the end of the analysis vector with the current
+ stack augmented with an entry for `unknown terminal' with a start
+ offset of the remembered offset and an end offset of the
+ current offset.
+@end itemize
+@end table
+
+@deffn {Generic Function} count-check nonterminal-entry
+
+A count check means that we compare the expression count of the top of
+the stack and its syntactic category. If the category has a limited
+number of expressions allowed, and we have reached it, we pop off the
+top element, increment the expression count of the new top entry, and
+perform a count check again.
+@end deffn
@chapter The undo protocol
