[slime-devel] [investigation] Font lock, reader macros and SLIME repl
Hello SLIME developers, Both in the lisp-mode and in slime REPL, there is currently one major thing that Emacs isn't getting right: constructs using reader macros like #P"/home/" and #2A((1 0) (0 1)) are regarded as two sexps instead of one. E.g. command for wrapping a sexp may turn #P"/home/" into (#P)"/home/", etc. It gets even more awful with paredit-mode (that I prefer to use), but the problem is noticeable even without it. I have investigated one possible solution to it, and here are the results. First, I decided to limit the scope of solution to some predefined "shape" of reader macro: for starters, when #\# is used as dispatch macro character, and its reader function calls READ to get the sexp following #[numarg]<char>. This is the case with all standard reader-macros and many implementation-specific and user-defined ones. Furthermore, I discarded the possibility to consult SWANK for the buffer's *readtable* (SWANK doesn't *know* it, anyway). To get right behavior for #P, #2A and the like, the most appropriate syntax for all macro-characters seems to be `expression prefix': we want Emacs to regard the whole #P thing as it would regard a quote, backquote or comma. Let's use the feature of font-lock mode in modern emacsen: syntactic keywords (in a nutshell, it's when a code used for text-coloring is [ab]used to add syntax-table properties to some text fragments). `parse-sexp-lookup-properties' set to true causes the sexp machinery to look into syntax-table properties of individual characters. Let's add a syntactic keyword to lisp-mode: (defun aak:add-lisp-reader-macros-syntactic-keyword () "Register # numarg macro-character as a font lock syntactic keyword, turning it into expression prefix." (set (make-local-variable 'parse-sexp-lookup-properties) t) (set (make-local-variable 'font-lock-syntactic-keywords) '(("\\(\\W\\|$\\)\\#\\([0-9]*[A-Za-z]\\)" (2 "'"))))) (add-hook 'lisp-mode-hook 'aak:add-lisp-reader-macros-syntactic-keyword) With the hook above, any lisp-mode buffer with font-lock mode enabled gets good enough navigation/wrapping/unwrapping for previously misinterpreted items. I decided to leave macro-characters handled by this regexp to [A-Za-z], because normal behavior of reading the following sexp can't really be expected from other characters: readtable additions along the lines of #"something" and #{something} are also popular, and we can't get _them_ right without knowing a concrete syntaxt they expect. Another decision that has its merits is to limit it further to _standard_ macro-characters only. Now I wanted the same thing to be active for slime-repl mode. The only obstacle was that slime-repl mode is currently font-lock-unfriendly: when some faces are set programmatically and some left to font-lock, the programmatically-set faces should use font-lock-face property instead of face property (the latter gets overridden by font-lock results). A workaround that I use currently is the following advice to add-text-properties: (defadvice add-text-properties (before aak:slime-propertize-with-font-lock-face activate) (when (eq 'slime-repl-mode major-mode) (let* ((props (ad-get-arg 2)) (face (getf props 'face))) (when face (push face props) (push 'font-lock-face props) (when (memq 'face (getf props 'rear-nonsticky)) (push 'font-lock-face (getf props 'rear-nonsticky))) (ad-set-arg 2 props))))) When in slime-repl mode, it copies face property to font-lock-face property, and also makes it rear-nonsticky when face is requested to be. It's the only part that really touches SLIME, and I think it would better insert font-lock-face property itself, together with face property (don't forget rear-nonstickyness). This duplication (face => font-lock-face) does no harm when font-lock mode is disabled, but enables slime-repl buffer to be used with font-lock enabled as well. Another hook to enable font-lock mode in slime buffers: (defun aak:enable-font-lock-for-slime-repl () "Enable font lock mode in slime REPL buffer." (font-lock-fontify-buffer) ;; don't know why it's needed.. (font-lock-mode 1)) (add-hook 'slime-repl-mode-hook 'aak:add-lisp-reader-macros-syntactic-keyword) (add-hook 'slime-repl-mode-hook 'aak:enable-font-lock-for-slime-repl) NB. The problem that I'm trying to solve is even more important for slime-repl buffers, where incorrect sexp wrapping tends to break _presentations_ apart. -- Regards, Anton Kovalenko +7(916)345-34-02 | Elektrostal' MO, Russia
In article <87pqx0kmga.fsf@desktop-ng.home.sw4me.com>, Anton Kovalenko <anton@sw4me.com> wrote:
Hello SLIME developers,
Both in the lisp-mode and in slime REPL, there is currently one major thing that Emacs isn't getting right: constructs using reader macros like #P"/home/" and #2A((1 0) (0 1)) are regarded as two sexps instead of one. E.g. command for wrapping a sexp may turn #P"/home/" into (#P)"/home/", etc. It gets even more awful with paredit-mode (that I prefer to use), but the problem is noticeable even without it.
I have investigated one possible solution to it, and here are the results.
First, I decided to limit the scope of solution to some predefined "shape" of reader macro: for starters, when #\# is used as dispatch macro character, and its reader function calls READ to get the sexp following #[numarg]<char>. This is the case with all standard reader-macros and many implementation-specific and user-defined ones. Furthermore, I discarded the possibility to consult SWANK for the buffer's *readtable* (SWANK doesn't *know* it, anyway).
To get right behavior for #P, #2A and the like, the most appropriate syntax for all macro-characters seems to be `expression prefix': we want Emacs to regard the whole #P thing as it would regard a quote, backquote or comma.
Let's use the feature of font-lock mode in modern emacsen: syntactic keywords (in a nutshell, it's when a code used for text-coloring is [ab]used to add syntax-table properties to some text fragments). `parse-sexp-lookup-properties' set to true causes the sexp machinery to look into syntax-table properties of individual characters.
Let's add a syntactic keyword to lisp-mode:
(defun aak:add-lisp-reader-macros-syntactic-keyword () "Register # numarg macro-character as a font lock syntactic keyword, turning it into expression prefix." (set (make-local-variable 'parse-sexp-lookup-properties) t) (set (make-local-variable 'font-lock-syntactic-keywords) '(("\\(\\W\\|$\\)\\#\\([0-9]*[A-Za-z]\\)" (2 "'")))))
(add-hook 'lisp-mode-hook 'aak:add-lisp-reader-macros-syntactic-keyword)
Yes, I've been using (defun enable-cl-syntax () (make-local-variable 'font-lock-syntactic-keywords) (set (make-local-variable 'parse-sexp-lookup-properties) t) (let ((regexp (regexp-opt '("#*" "#." "#=" "#A" "#C" "#P" "#S")))) (pushnew `(,regexp 0 "'") font-lock-syntactic-keywords :test #'equal))) (add-hook 'lisp-mode-hook #'enable-cl-syntax) for a long time now.
With the hook above, any lisp-mode buffer with font-lock mode enabled gets good enough navigation/wrapping/unwrapping for previously misinterpreted items.
I decided to leave macro-characters handled by this regexp to [A-Za-z], because normal behavior of reading the following sexp can't really be expected from other characters: readtable additions along the lines of #"something" and #{something} are also popular, and we can't get _them_ right without knowing a concrete syntaxt they expect. Another decision that has its merits is to limit it further to _standard_ macro-characters only.
There could be a SWANK:DECLAIM-SYNTAX along SWANK:DECLAIM-INDENTATION which people could put into their CL source files. -T.
Tobias C Rittweiler <tcr@freebits.de> writes:
There could be a SWANK:DECLAIM-SYNTAX along SWANK:DECLAIM-INDENTATION which people could put into their CL source files.
While we're at it, there could also be a symbol pushed to *features*, on which these declarations would be conditionalized (we don't want to _depend_ on swank unconditionally, do we?). Adapted from my .sbclrc: (when (find-package "SWANK") (pushnew :SWANK *features*)) .... #+swank (do something useful for swank/slime) -- Regards, Anton Kovalenko +7(916)345-34-02 | Elektrostal' MO, Russia
In article <87iq2rkqeq.fsf@desktop-ng.home.sw4me.com>, Anton Kovalenko <anton@sw4me.com> wrote:
Tobias C Rittweiler <tcr@freebits.de> writes:
There could be a SWANK:DECLAIM-SYNTAX along SWANK:DECLAIM-INDENTATION which people could put into their CL source files.
While we're at it, there could also be a symbol pushed to *features*, on which these declarations would be conditionalized (we don't want to _depend_ on swank unconditionally, do we?).
Adapted from my .sbclrc:
(when (find-package "SWANK") (pushnew :SWANK *features*)) .... #+swank (do something useful for swank/slime)
Yes I had the same issue before. Logged as https://bugs.launchpad.net/slime/+bug/627313 -T.
On Aug 31, 2010, at 2:18 AM, Tobias C Rittweiler wrote:
In article <87pqx0kmga.fsf@desktop-ng.home.sw4me.com>, Anton Kovalenko <anton@sw4me.com> wrote:
Hello SLIME developers,
Both in the lisp-mode and in slime REPL, there is currently one major thing that Emacs isn't getting right: constructs using reader macros like #P"/home/" and #2A((1 0) (0 1)) are regarded as two sexps instead of one. E.g. command for wrapping a sexp may turn #P"/home/" into (#P)"/home/", etc. It gets even more awful with paredit-mode (that I prefer to use), but the problem is noticeable even without it.
I have investigated one possible solution to it, and here are the results.
I recall we've discussed this issue in the past. Rather than making emacs (as swank client) recognize the bounds of the lisp expression, swank could cover this functionality on the server. That is, emacs sends the expression to the swank server and gets back information about the boundaries of the expression. For example, in the case of a reader macro: 1. Emacs streams the reader macro as characters to the swank server, continuing beyond the end of the expression. 2. The swank server parses from the stream until the reader reaches the end of the expression. 3. The server notifies emacs via swank how many characters were read when reaching the end of the expression. 4. Emacs uses the information about how many characters were read to determine when the expression ended. Using the inferior lisp to determine expression bounds has several benefits, including: 1. Emacs doesn't have to be configured to know the specific lisp syntax. 2. Clojure, Scheme and other lisp dialects can be supported with no fuss. 3. SLIME can handle custom reader macros with ease. 4. No additional configuration required to cover custom reader macros. -- Terje
First, I decided to limit the scope of solution to some predefined "shape" of reader macro: for starters, when #\# is used as dispatch macro character, and its reader function calls READ to get the sexp following #[numarg]<char>. This is the case with all standard reader-macros and many implementation-specific and user-defined ones. Furthermore, I discarded the possibility to consult SWANK for the buffer's *readtable* (SWANK doesn't *know* it, anyway).
To get right behavior for #P, #2A and the like, the most appropriate syntax for all macro-characters seems to be `expression prefix': we want Emacs to regard the whole #P thing as it would regard a quote, backquote or comma.
Let's use the feature of font-lock mode in modern emacsen: syntactic keywords (in a nutshell, it's when a code used for text-coloring is [ab]used to add syntax-table properties to some text fragments). `parse-sexp-lookup-properties' set to true causes the sexp machinery to look into syntax-table properties of individual characters.
Let's add a syntactic keyword to lisp-mode:
(defun aak:add-lisp-reader-macros-syntactic-keyword () "Register # numarg macro-character as a font lock syntactic keyword, turning it into expression prefix." (set (make-local-variable 'parse-sexp-lookup-properties) t) (set (make-local-variable 'font-lock-syntactic-keywords) '(("\\(\\W\\|$\\)\\#\\([0-9]*[A-Za-z]\\)" (2 "'")))))
(add-hook 'lisp-mode-hook 'aak:add-lisp-reader-macros-syntactic-keyword)
Yes, I've been using
(defun enable-cl-syntax () (make-local-variable 'font-lock-syntactic-keywords) (set (make-local-variable 'parse-sexp-lookup-properties) t) (let ((regexp (regexp-opt '("#*" "#." "#=" "#A" "#C" "#P" "#S")))) (pushnew `(,regexp 0 "'") font-lock-syntactic-keywords :test #'equal)))
(add-hook 'lisp-mode-hook #'enable-cl-syntax)
for a long time now.
With the hook above, any lisp-mode buffer with font-lock mode enabled gets good enough navigation/wrapping/unwrapping for previously misinterpreted items.
I decided to leave macro-characters handled by this regexp to [A-Za-z], because normal behavior of reading the following sexp can't really be expected from other characters: readtable additions along the lines of #"something" and #{something} are also popular, and we can't get _them_ right without knowing a concrete syntaxt they expect. Another decision that has its merits is to limit it further to _standard_ macro-characters only.
There could be a SWANK:DECLAIM-SYNTAX along SWANK:DECLAIM-INDENTATION which people could put into their CL source files.
-T.
_______________________________________________ slime-devel site list slime-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/slime-devel
In article <4818B0D6-5407-4B36-A2B7-1F50450DEA5F@in-progress.com>, Terje Norderhaug <terje@in-progress.com> wrote:
On Aug 31, 2010, at 2:18 AM, Tobias C Rittweiler wrote:
In article <87pqx0kmga.fsf@desktop-ng.home.sw4me.com>, Anton Kovalenko <anton@sw4me.com> wrote:
Hello SLIME developers,
Both in the lisp-mode and in slime REPL, there is currently one major thing that Emacs isn't getting right: constructs using reader macros like #P"/home/" and #2A((1 0) (0 1)) are regarded as two sexps instead of one. E.g. command for wrapping a sexp may turn #P"/home/" into (#P)"/home/", etc. It gets even more awful with paredit-mode (that I prefer to use), but the problem is noticeable even without it.
I have investigated one possible solution to it, and here are the results.
I recall we've discussed this issue in the past.
Rather than making emacs (as swank client) recognize the bounds of the lisp expression, swank could cover this functionality on the server. That is, emacs sends the expression to the swank server and gets back information about the boundaries of the expression.
For example, in the case of a reader macro:
1. Emacs streams the reader macro as characters to the swank server, continuing beyond the end of the expression. 2. The swank server parses from the stream until the reader reaches the end of the expression. 3. The server notifies emacs via swank how many characters were read when reaching the end of the expression. 4. Emacs uses the information about how many characters were read to determine when the expression ended.
Using the inferior lisp to determine expression bounds has several benefits, including:
1. Emacs doesn't have to be configured to know the specific lisp syntax. 2. Clojure, Scheme and other lisp dialects can be supported with no fuss. 3. SLIME can handle custom reader macros with ease. 4. No additional configuration required to cover custom reader macros.
You mean for every time I press C-M-f? Is that how your editor thingie works? -T.
On Sep 4, 2010, at 1:34 PM, Tobias C Rittweiler wrote:
In article <4818B0D6-5407-4B36-A2B7-1F50450DEA5F@in-progress.com>, Terje Norderhaug <terje@in-progress.com> wrote:
On Aug 31, 2010, at 2:18 AM, Tobias C Rittweiler wrote:
In article <87pqx0kmga.fsf@desktop-ng.home.sw4me.com>, Anton Kovalenko <anton@sw4me.com> wrote:
Hello SLIME developers,
Both in the lisp-mode and in slime REPL, there is currently one major thing that Emacs isn't getting right: constructs using reader macros like #P"/home/" and #2A((1 0) (0 1)) are regarded as two sexps instead of one. E.g. command for wrapping a sexp may turn #P"/home/" into (#P)"/home/", etc. It gets even more awful with paredit-mode (that I prefer to use), but the problem is noticeable even without it.
I have investigated one possible solution to it, and here are the results.
I recall we've discussed this issue in the past.
Rather than making emacs (as swank client) recognize the bounds of the lisp expression, swank could cover this functionality on the server. That is, emacs sends the expression to the swank server and gets back information about the boundaries of the expression.
For example, in the case of a reader macro:
1. Emacs streams the reader macro as characters to the swank server, continuing beyond the end of the expression. 2. The swank server parses from the stream until the reader reaches the end of the expression. 3. The server notifies emacs via swank how many characters were read when reaching the end of the expression. 4. Emacs uses the information about how many characters were read to determine when the expression ended.
Using the inferior lisp to determine expression bounds has several benefits, including:
1. Emacs doesn't have to be configured to know the specific lisp syntax. 2. Clojure, Scheme and other lisp dialects can be supported with no fuss. 3. SLIME can handle custom reader macros with ease. 4. No additional configuration required to cover custom reader macros.
You mean for every time I press C-M-f? Is that how your editor thingie works?
I am particularly concerned about lisp developers consistently being able to evaluate the current expression also when it contains reader macros and custom syntax (including other lisp dialects). As to your question, like emacs, MCLIDE uses "rules of thumb" strategies to match balanced expressions. In fact, the lip editor of MCLIDE is a modernized version of good old Fred ("Fred Resembles Emacs Deliberately"), a Common Lisp implementation of relevant emacs lisp editor functionality. So C-M-f and other balance matching shortcuts in MCLIDE currently have similar shortcomings as emacs SLIME. Terje Norderhaug terje@in-progress.com
Terje Norderhaug <terje@in-progress.com> writes:
On Sep 4, 2010, at 1:34 PM, Tobias C Rittweiler wrote:
You mean for every time I press C-M-f? Is that how your editor thingie works?
I am particularly concerned about lisp developers consistently being able to evaluate the current expression also when it contains reader macros and custom syntax (including other lisp dialects).
For what it's worth, I would very much like this too. For example, I the logic to find the operator appropriate to a given position is fine for lisps with standard syntax, but not so good for (for example) R -- which means that the logic for looking up arglists gets the wrong answer. There's of course a tasteful balance to be struck between hard-coding (which is fine for most normal situations, in any case) and excessive querying of the arbitrarily remote "lisp"... Cheers, Christophe
participants (4)
-
Anton Kovalenko -
Christophe Rhodes -
Terje Norderhaug -
Tobias C Rittweiler