[slime-devel] broken clojure REPL
FWIW, the following patch broke the clojure REPL (*inferior-lisp* still works, but not the fancy REPL): commit addc5be70a489cdc8ed555a2fed7273040da196e Author: Tobias C. Rittweiler <trittweiler@common-lisp.net> Date: Sat Oct 31 22:13:55 2009 +0000 * slime.el (slime-inside-string-p, slime-inside-comment-p) (slime-inside-string-or-comment-p): New. * swank-match.lisp: New file. Contains very simple pattern matcher from the CMU AI archive. * swank-loader.lisp: Compile swank-match.lisp. * swank.lisp: Make SWANK package use new SWANK-MATCH package. * slime-autodoc.el, swank-arglists.lisp: Large parts were rewritten. Autodoc is now able to highlight &key parameters, and parameters in nested arglists. * slime-parse.el, slime-c-p-c.el, slime-highlighting-edits.el: Adapted to changes. the symptom is the following: ; SLIME 2009-10-31 user> 3 3 user> (+ 3 4) at which point it just hangs... nothing (helpful) in *messages* or *inferior-lisp*. Reverting to the previous commit fixes the problem. It's really nice to be able to use git to quickly bisect the history to find the offending commit. Not so nice when that commit is >2k lines :( TCR, any ideas? thanks, cyrus
Cyrus Harmon <ch-slime@bobobeach.com> writes:
FWIW, the following patch broke the clojure REPL (*inferior-lisp* still works, but not the fancy REPL):
commit addc5be70a489cdc8ed555a2fed7273040da196e Author: Tobias C. Rittweiler <trittweiler@common-lisp.net> Date: Sat Oct 31 22:13:55 2009 +0000
* slime.el (slime-inside-string-p, slime-inside-comment-p) (slime-inside-string-or-comment-p): New.
* swank-match.lisp: New file. Contains very simple pattern matcher from the CMU AI archive.
* swank-loader.lisp: Compile swank-match.lisp.
* swank.lisp: Make SWANK package use new SWANK-MATCH package.
* slime-autodoc.el, swank-arglists.lisp: Large parts were rewritten. Autodoc is now able to highlight &key parameters, and parameters in nested arglists.
* slime-parse.el, slime-c-p-c.el, slime-highlighting-edits.el: Adapted to changes.
the symptom is the following:
; SLIME 2009-10-31 user> 3 3 (+ 3 4)
at which point it just hangs... nothing (helpful) in *messages* or *inferior-lisp*. Reverting to the previous commit fixes the problem. It's really nice to be able to use git to quickly bisect the history to find the offending commit. Not so nice when that commit is
2k lines :(
TCR, any ideas?
I'm sorry I don't know. I thought the Clojure people forked Slime, and presumably would reintegrate upstream changes as their time allows. But you seem to be bitten by updating your CVS checkout. Does the clojure stuff provide an implementation of the necessary backend for autodoc? If so, the API changed. If not, do not use the slime-autodoc contrib with clojure. What are the last rpc expressions in *slime-events*? -T.
Hi, I've been bitten by the same a few days ago. -------- Original-Nachricht --------
Datum: Sun, 15 Nov 2009 10:08:21 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
Cyrus Harmon <ch-slime@bobobeach.com> writes:
FWIW, the following patch broke the clojure REPL (*inferior-lisp* still works, but not the fancy REPL):
[...]
the symptom is the following:
; SLIME 2009-10-31 user> 3 3 (+ 3 4)
at which point it just hangs... nothing (helpful) in *messages* or *inferior-lisp*. Reverting to the previous commit fixes the problem. It's really nice to be able to use git to quickly bisect the history to find the offending commit. Not so nice when that commit is
2k lines :(
TCR, any ideas?
I'm sorry I don't know. I thought the Clojure people forked Slime, and presumably would reintegrate upstream changes as their time allows.
I ask myself, how many Clojure users are reading this list and are using Slime from CVS? Maybe enough to consider integrating official support for Clojure?
But you seem to be bitten by updating your CVS checkout.
Does the clojure stuff provide an implementation of the necessary backend for autodoc? If so, the API changed. If not, do not use the slime-autodoc contrib with clojure.
What are the last rpc expressions in *slime-events*?
(:emacs-rex (swank:connection-info) "COMMON-LISP-USER" t 1) (:indentation-update (("binding-map" . 1) ("with-pretty-writer" . 1) ("with-pprint-dispatch" . 1) ("dothread-keeping-clj" . 1) ("dothread-keeping" . 1) ("dothread" . 0) ("with-connection" . 1) ("with-bindings" . 0) ("with-emacs-package" . 0) ("dothread-swank" . 0) ("with-package-tracking" . 0) ("doseq" . 1) ("letfn" . 1) ("cond" . 0) ("with-open" . 1) ("sync" . 1) ("let" . 1) ("dotimes" . 1) ("with-in-str" . 1) ("loop" . 1) ...)) (:return (:ok (:pid "506" :style :spawn :lisp-implementation (:type "clojure" :name "clojure") :package (:name "user" :prompt "user") :version "2009-11-05")) 1) (:emacs-rex (swank:swank-require '(:swank-package-fu :swank-fuzzy :swank-fancy-inspector :swank-arglists)) "COMMON-LISP-USER" t 2) (:emacs-rex (swank:swank-require :swank-presentations) "COMMON-LISP-USER" t 3) (:emacs-rex (swank:create-repl nil) "COMMON-LISP-USER" t 4) (:return (:ok ("user" "user")) 4) (:return (:ok nil) 3) (:indentation-update (("with-timeout" . 1))) (:return (:ok nil) 2) (:emacs-rex (swank:listener-eval "3\n") "user" :repl-thread 5) (:write-string "3\n" :repl-result) (:return (:ok nil) 5) (:emacs-rex (swank:arglist-for-echo-area '("+" "" swank::%cursor-marker%) :print-right-margin 1000 :print-lines 1) "user" :repl-thread 6) (:emacs-rex (swank:arglist-for-echo-area '("+" "3" "4" swank::%cursor-marker%) :print-right-margin 1000 :print-lines 1) "user" :repl-thread 7) (:emacs-rex (swank:listener-eval "(+ 3 4)\n") "user" :repl-thread 8) Hope that helps. Kind regards, Stefan -- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser
"Stefan Kamphausen" <skampi@gmx.net> writes:
I ask myself, how many Clojure users are reading this list and are using Slime from CVS? Maybe enough to consider integrating official support for Clojure?
Non-CL backends, like contribs, are maintained by whoever feels like it. There's no "official support". I wouldn't mind distributing a swank backend for clojure within the contrib/ directory -- but, personally, I wouldn't have bothered checking its workingness back then even if such a backend had been available in the official distribution.
What are the last rpc expressions in *slime-events*?
... (:emacs-rex (swank:arglist-for-echo-area '("+" "" swank::%cursor-marker%) :print-right-margin 1000 :print-lines 1) "user" :repl-thread 6) (:emacs-rex (swank:arglist-for-echo-area '("+" "3" "4" swank::%cursor-marker%) :print-right-margin 1000 :print-lines 1) "user" :repl-thread 7) (:emacs-rex (swank:listener-eval "(+ 3 4)\n") "user" :repl-thread 8)
The API for SWANK:ARGLIST-FOR-ECHO-AREA changed; I do not know why that change triggers an infinite loop in the respective clojure code. My hands are tied on this issue -- report to the maintainers of the clojure fork that this portion of the API changed. The new API is described in a comment in swank-arglists.lisp. (The upshot is that the new API allows for more fined-grained highlighting of parameters in case Clojure has something like &key parameters.) -T.
Hi, -------- Original-Nachricht --------
Datum: Sun, 15 Nov 2009 14:39:48 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
"Stefan Kamphausen" <skampi@gmx.net> writes:
I ask myself, how many Clojure users are reading this list and are using Slime from CVS? Maybe enough to consider integrating official support for Clojure?
Non-CL backends, like contribs, are maintained by whoever feels like it. There's no "official support". I wouldn't mind distributing a swank backend for clojure within the contrib/ directory -- but, personally, I wouldn't have bothered checking its workingness back then even if such a backend had been available in the official distribution.
I raised the topic on the Clojure discussion group. Thank you for providing background information. Kind regards, Stefan -- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser
Hi again, -------- Original-Nachricht --------
Datum: Sun, 15 Nov 2009 15:22:54 +0100 Von: "Stefan Kamphausen" <skampi@gmx.net> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
I raised the topic on the Clojure discussion group. Thank you for providing background information.
after that I tried to understand what's happening behind the scenes. And now I'd like to ask some questions. First and most important to me is the line (defconst slime-cursor-marker 'swank::%cursor-marker%) from contrib/slime-parse.el. If I tracked things correctly this symbol gets inserted in the form sent to swank, which is passed to arglist-for-echo-area by the swank-evaluation mechanism. This seems to rely on the fact that "swank::%cursor-marker%" is a valid name in the backend. Surely, this is true for CL-backends, unfortunately not for Clojure. There "::" should be a "/" and the "%"s are not allowed. Please see http://groups.google.com/group/clojure/t/eef10af52a619eee for a thread on the Clojure-list. While I understand that it would be nice to have a real symbol denoting the current cursor position, it feels rather hackish to me (and it /kills/ the swank-threads in Clojure). Would you mind replacing it with a not-so-special string? As far as I can tell, the code wouldn't need to change too much (cases of eq would probably become string= or something like that in a few places, but I may be wrong). I managed to get the backend working again, well, kind of. After not crashing anymore, I could analyze the new way forms get sent over the wire and changed a few functions. What remains then is an error message from eldoc, e.g. eldoc error: (wrong-type-argument listp [string file line)) I still have to find the piece of code that throws this, but it seems to be related to braces and brackets, some code seems to rely on parens being the only kind of parenthesis. Kind regards, Stefan -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
"Stefan Kamphausen" <skampi@gmx.net> writes:
First and most important to me is the line
(defconst slime-cursor-marker 'swank::%cursor-marker%)
from contrib/slime-parse.el. If I tracked things correctly this symbol gets inserted in the form sent to swank, which is passed to arglist-for-echo-area by the swank-evaluation mechanism. This seems to rely on the fact that "swank::%cursor-marker%" is a valid name in the backend. Surely, this is true for CL-backends, unfortunately not for Clojure. There "::" should be a "/" and the "%"s are not allowed. Please see http://groups.google.com/group/clojure/t/eef10af52a619eee for a thread on the Clojure-list.
While I understand that it would be nice to have a real symbol denoting the current cursor position, it feels rather hackish to me (and it /kills/ the swank-threads in Clojure). Would you mind replacing it with a not-so-special string? As far as I can tell, the code wouldn't need to change too much (cases of eq would probably become string= or something like that in a few places, but I may be wrong).
Using true symbols feels hackish to you but using strings which may legitimately appear as buffer forms does not? Uhm. :-) Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
What remains then is an error message from eldoc, e.g.
eldoc error: (wrong-type-argument listp [string file line))
I still have to find the piece of code that throws this, but it seems to be related to braces and brackets, some code seems to rely on parens being the only kind of parenthesis.
Backtrace? You have to make sure that ?\[ has a syntax-type "(", and ?\] a syntax-type ")" by using `modify-syntax-entry'. -T.
Hi Tobias, -------- Original-Nachricht --------
Datum: Tue, 17 Nov 2009 21:04:21 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
"Stefan Kamphausen" <skampi@gmx.net> writes: [...]
While I understand that it would be nice to have a real symbol denoting the current cursor position, it feels rather hackish to me (and it /kills/ the swank-threads in Clojure). Would you mind replacing it with a not-so-special string? As far as I can tell, the code wouldn't need to change too much (cases of eq would probably become string= or something like that in a few places, but I may be wrong).
Using true symbols feels hackish to you but using strings which may legitimately appear as buffer forms does not? Uhm. :-)
Of course your are right. Having something definitive like a symbol is way better than a not-so-special string like "*HERE*" or something. The thing that feels strange to me is to create a symbol in elisp which uses the naming rules from common-lisp. Maybe another symbol name would be possible, hard for me to say, especially when dealing with packages/namespaces. A string is just a quick and easy solution, without being as robust as a symbol.
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
What remains then is an error message from eldoc, e.g.
eldoc error: (wrong-type-argument listp [string file line))
I still have to find the piece of code that throws this, but it seems to be related to braces and brackets, some code seems to rely on parens being the only kind of parenthesis.
Backtrace?
I can't get one, debug-on-error doesn't give it and the function debug-on-signal, which I seem to remember from Emacs 22.x seems to have gone in my 23.1 version. Uch, another focus. You have to make sure that ?\[ has a syntax-type "(", and ?\]
a syntax-type ")" by using `modify-syntax-entry'.
Clojure-mode looks save here: (defvar clojure-mode-syntax-table (let ((table (copy-syntax-table emacs-lisp-mode-syntax-table))) (modify-syntax-entry ?~ "' " table) (modify-syntax-entry ?, " " table) (modify-syntax-entry ?\{ "(}" table) (modify-syntax-entry ?\} "){" table) (modify-syntax-entry ?\[ "(]" table) (modify-syntax-entry ?\] ")[" table) (modify-syntax-entry ?^ "'" table) table)) Thank your for taking your time to help with this, although it's not your backend that's in trouble. Best, Stefan -- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser
"Stefan Kamphausen" <skampi@gmx.net> writes:
Hi Tobias,
-------- Original-Nachricht --------
Datum: Tue, 17 Nov 2009 21:04:21 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
"Stefan Kamphausen" <skampi@gmx.net> writes: [...]
While I understand that it would be nice to have a real symbol denoting the current cursor position, it feels rather hackish to me (and it /kills/ the swank-threads in Clojure). Would you mind replacing it with a not-so-special string? As far as I can tell, the code wouldn't need to change too much (cases of eq would probably become string= or something like that in a few places, but I may be wrong). Using true symbols feels hackish to you but using strings which may legitimately appear as buffer forms does not? Uhm. :-)
Of course your are right. Having something definitive like a symbol is way better than a not-so-special string like "*HERE*" or something. The thing that feels strange to me is to create a symbol in elisp which uses the naming rules from common-lisp.
Elisp is pretty liberal on what it allows as symbol names.
Maybe another symbol name would be possible, hard for me to say, especially when dealing with packages/namespaces. A string is just a quick and easy solution, without being as robust as a symbol.
You didn't answer the important question:
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
I can change %cursor-marker% to have a symbol name that's nicer for clojure to cope with. But it would be an interim solution, only. I really want to see this fixed in Clojure. At the moment, the Clojure backend does not correctly implement parsing the protocol. -T.
Hi,
You didn't answer the important question:
that's because I'm not sure.
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
I'm not sure what exactly you mean in the first place. Are you referring to CL's "|" around names? Please give me an explanation.
I can change %cursor-marker% to have a symbol name that's nicer for clojure to cope with. But it would be an interim solution, only. I really want to see this fixed in Clojure.
At the moment, the Clojure backend does not correctly implement parsing the protocol.
Currently to me it looks like we can't get past the reader and then it's hard to implement some parsing. But again, I think I misunderstand what you are patiently trying to explain to me. FWIW, here's a very short doc taken from http://clojure.org/reader "Symbols begin with a non-numeric character and can contain alphanumeric characters and *, +, !, -, _, and ? (other characters will be allowed eventually, but not all macro characters have been determined). [...] A symbol can contain one or more non-repeating ':'s" Kind regards, Stefan -- DSL-Preisknaller: DSL Komplettpakete von GMX schon für 16,99 Euro mtl.!* Hier klicken: http://portal.gmx.net/de/go/dsl02
"Stefan Kamphausen" <skampi@gmx.net> writes:
Hi,
You didn't answer the important question:
that's because I'm not sure.
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
I'm not sure what exactly you mean in the first place. Are you referring to CL's "|" around names? Please give me an explanation.
Is there an INTERN function? Does it accept arbitrary strings?
I can change %cursor-marker% to have a symbol name that's nicer for clojure to cope with. But it would be an interim solution, only. I really want to see this fixed in Clojure.
At the moment, the Clojure backend does not correctly implement parsing the protocol.
Currently to me it looks like we can't get past the reader and then it's hard to implement some parsing.
But again, I think I misunderstand what you are patiently trying to explain to me.
FWIW, here's a very short doc taken from http://clojure.org/reader
"Symbols begin with a non-numeric character and can contain alphanumeric characters and *, +, !, -, _, and ? (other characters will be allowed eventually, but not all macro characters have been determined). [...] A symbol can contain one or more non-repeating :'s"
The clojure backend should parse the input itself, and pass the symbol name over to INTERN. -T.
Hi, just to not disappoint anyone. It'll take some time (probably some days) before I can look at this again. Cheers, Stefan -------- Original-Nachricht --------
Datum: Fri, 20 Nov 2009 15:21:26 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
"Stefan Kamphausen" <skampi@gmx.net> writes:
Hi,
You didn't answer the important question:
that's because I'm not sure.
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
I'm not sure what exactly you mean in the first place. Are you referring to CL's "|" around names? Please give me an explanation.
Is there an INTERN function? Does it accept arbitrary strings?
I can change %cursor-marker% to have a symbol name that's nicer for clojure to cope with. But it would be an interim solution, only. I really want to see this fixed in Clojure.
At the moment, the Clojure backend does not correctly implement parsing the protocol.
Currently to me it looks like we can't get past the reader and then it's hard to implement some parsing.
But again, I think I misunderstand what you are patiently trying to explain to me.
FWIW, here's a very short doc taken from http://clojure.org/reader
"Symbols begin with a non-numeric character and can contain alphanumeric characters and *, +, !, -, _, and ? (other characters will be allowed eventually, but not all macro characters have been determined). [...] A symbol can contain one or more non-repeating :'s"
The clojure backend should parse the input itself, and pass the symbol name over to INTERN.
-T.
_______________________________________________ slime-devel site list slime-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/slime-devel
-- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser
"Tobias C. Rittweiler" <tcr@freebits.de> writes:
The clojure backend should parse the input itself, and pass the symbol name over to INTERN.
It sounds like the problem here is that the Clojure Swank backend is parsing forms received using `read', and it's choking on reading symbols that violate its syntax. Can someone confirm that that's the case? If so, it's a shame to not be able to use a Lisp reader to read what look very much like Lisp forms. Studying the Clojure reader documentation¹, there's no mention a symbol escape syntax. As a compromise, would it be possible to have the Swank backend initialize this cursor treatment by /sending/ a symbol to the SLIME client, which the client would have for that session and use instead of the symbol swank::%cursor-marker%? Instead of being a constant, the current slime-cursor-marker could be provided as an argument to `slime-parse-form-upto-point'. Of course, it's possible that the backend could provide a symbol that isn't valid in ELisp, which just turns the problem around. Footnotes: ¹ http://clojure.org/reader -- Steven E. Harris
"Steven E. Harris" <seh@panix.com> writes:
"Tobias C. Rittweiler" <tcr@freebits.de> writes:
The clojure backend should parse the input itself, and pass the symbol name over to INTERN.
It sounds like the problem here is that the Clojure Swank backend is parsing forms received using `read', and it's choking on reading symbols that violate its syntax. Can someone confirm that that's the case?
If so, it's a shame to not be able to use a Lisp reader to read what look very much like Lisp forms. Studying the Clojure reader documentation¹, there's no mention a symbol escape syntax.
The point is that you're not trying to use a Lisp reader to read what looks very much like Lisp forms, but you're trying to use the Clojure reader. And the Clojure reader is very restrictive in what it accepts as symbols.
As a compromise, would it be possible to have the Swank backend initialize this cursor treatment by /sending/ a symbol to the SLIME client, which the client would have for that session and use instead of the symbol swank::%cursor-marker%?
Instead of being a constant, the current slime-cursor-marker could be provided as an argument to `slime-parse-form-upto-point'. Of course, it's possible that the backend could provide a symbol that isn't valid in ELisp, which just turns the problem around.
The easiest compromise is to change %CURSOR-MARKER% to +CURSOR-MARKER+ which, if I read the documentation right, the Clojure reader will accept. Doing so, is sweeping a problem under the carpet. I still think that it is Clojure that should "change". It should provide a way to READ in a form with minimal processing -- something that CL's reader, for instance, does not really allow to unfortunate consequences. It is useful to be able to minimally READ in a possibly untrusted data file -- without processing of reader macros, without interning. You get back forms of numbers, symbols (whose symbol-name is exactly the original textual representation), and possibly other data types that you allow by enabling their reader macros. Has anyone brought that up on a Clojure devel list? -T.
"Tobias C. Rittweiler" <tcr@freebits.de> writes:
The point is that you're not trying to use a Lisp reader to read what looks very much like Lisp forms, but you're trying to use the Clojure reader. And the Clojure reader is very restrictive in what it accepts as symbols.
Well, I wrote "Lisp" as opposed to "Common Lisp" to generalize the complaint. One can imagine SLIME having originated for some Lisp that allowed symbols like ":::foo", which the Common Lisp reader would not read without complaint. Are we not biased in this debate due to SLIME having been made for Common Lisp?
The easiest compromise is to change %CURSOR-MARKER% to +CURSOR-MARKER+ which, if I read the documentation right, the Clojure reader will accept.
It would not accept the "swank::" prefix, due to the double colon, but the plus is accepted where the percent sign is not.
Doing so, is sweeping a problem under the carpet.
Yes, I agree, insofar as either protocol participant is using a parser (be it the Common Lisp, Clojure, or ELisp reader) that isn't just reading a string. However, to recommend that the protocol participants implement a parser from scratch to read these Lisp forms seems unreasonable too; after all, the protocol looks to have been designed for writing and reading Lisp forms directly. It can't just be an accident that there are paired parentheses and keyword-style tags.
I still think that it is Clojure that should "change". It should provide a way to READ in a form with minimal processing -- something that CL's reader, for instance, does not really allow to unfortunate consequences.
That would be convenient, but also isn't fair, in light of my suggestion above about a symbol like ":::foo". I don't have an example of a symbol that Clojure can read and CL can't, but if the situation were reversed and Swank had been implemented first in Clojure, would you advocate that CL should be changed to play along? Does using the protocol require a parser separate from a lowest-common-denominator Lisp reader? If so, would you introduce syntax that the Common Lisp reader couldn't accept?
It is useful to be able to minimally READ in a possibly untrusted data file -- without processing of reader macros, without interning. You get back forms of numbers, symbols (whose symbol-name is exactly the original textual representation), and possibly other data types that you allow by enabling their reader macros.
Has anyone brought that up on a Clojure devel list?
Not that I've seen, no. This subject came up again in the context of a thread¹ I joined on trying to get Clojure and SLIME to work together. Footnotes: ¹ http://groups.google.com/group/clojure/msg/5f3df67b4e88b91c -- Steven E. Harris
"Steven E. Harris" <seh@panix.com> writes:
"Tobias C. Rittweiler" <tcr@freebits.de> writes:
The point is that you're not trying to use a Lisp reader to read what looks very much like Lisp forms, but you're trying to use the Clojure reader. And the Clojure reader is very restrictive in what it accepts as symbols.
Well, I wrote "Lisp" as opposed to "Common Lisp" to generalize the complaint. One can imagine SLIME having originated for some Lisp that allowed symbols like ":::foo", which the Common Lisp reader would not read without complaint. Are we not biased in this debate due to SLIME having been made for Common Lisp?
The easiest compromise is to change %CURSOR-MARKER% to +CURSOR-MARKER+ which, if I read the documentation right, the Clojure reader will accept.
It would not accept the "swank::" prefix, due to the double colon, but the plus is accepted where the percent sign is not.
I think I was told that the issue of two colons is not insurmountable.
Doing so, is sweeping a problem under the carpet.
Yes, I agree, insofar as either protocol participant is using a parser (be it the Common Lisp, Clojure, or ELisp reader) that isn't just reading a string.
However, to recommend that the protocol participants implement a parser from scratch to read these Lisp forms seems unreasonable too; after all, the protocol looks to have been designed for writing and reading Lisp forms directly. It can't just be an accident that there are paired parentheses and keyword-style tags.
I have not made such a recommendation in my previous mail.
I still think that it is Clojure that should "change". It should provide a way to READ in a form with minimal processing -- something that CL's reader, for instance, does not really allow to unfortunate consequences.
That would be convenient, but also isn't fair, in light of my suggestion above about a symbol like ":::foo". I don't have an example of a symbol that Clojure can read and CL can't, but if the situation were reversed and Swank had been implemented first in Clojure, would you advocate that CL should be changed to play along?
CL should have been made to provide such a minimal reading mode, too. Clojure, unlike ANSI CL, has the chance to make it happen. Please try to make it happen. -T.
"Tobias C. Rittweiler" <tcr@freebits.de> writes:
I think I was told that the issue of two colons is not insurmountable.
I was just basing that prohibition on what I see in Clojure 1.1.0 today by evaluating ,---- | (read-string "swank::+foo+") `---- which yields ,---- | java.lang.Exception: Invalid token: swank::+foo+ `---- Note that the `symbol' function does allow constructing a symbol with the offending colons: ,---- | > (symbol "swank::+foo+") | swank::+foo+ `---- Again, though, the Clojure reader documentation warns: A symbol can contain one or more non-repeating ':'s. While the `symbol' function isn't enforcing that restriction, it looks like the /reader/ doesn't intend to accept such a symbol.
I have not made such a recommendation in my previous mail.
Then I misunderstood your intent. Rereading, is your proposal that there should be a reader mode that interprets characters only to resolve them to numbers and symbols? How about delimited strings?
CL should have been made to provide such a minimal reading mode, too.
Clojure, unlike ANSI CL, has the chance to make it happen.
Please try to make it happen.
I'm just coming to Clojure in the last couple of weeks (not wanting to dive in without SLIME), so I don't yet know how receptive the maintainer is to such suggestions. And first, I and others need to better understand what you're proposing. -- Steven E. Harris
"Steven E. Harris" <seh@panix.com> writes:
I have not made such a recommendation in my previous mail.
Then I misunderstood your intent. Rereading, is your proposal that there should be a reader mode that interprets characters only to resolve them to numbers and symbols? How about delimited strings?
Yes, and symbols are READ in very literally, as if -- in CL syntax -- vertical bars were put around them. (Neglecting the issue of readtable-case here as that's probably to appliable to Clojure anyway.) (let ((*read-minimal-mode* t)) (read-from-string "(:::foo #+ bar)")) => (|:::foo| |#+| |bar|) Reading of strings (and other data structures) depends on whether their reader macros are active. Hopefully that's orthogonal in Clojure, too, already. -T.
On Jan 9, 2010, at 7:23 AM, Steven E. Harris wrote:
"Tobias C. Rittweiler" <tcr@freebits.de> writes:
The point is that you're not trying to use a Lisp reader to read what looks very much like Lisp forms, but you're trying to use the Clojure reader. And the Clojure reader is very restrictive in what it accepts as symbols.
Well, I wrote "Lisp" as opposed to "Common Lisp" to generalize the complaint. One can imagine SLIME having originated for some Lisp that allowed symbols like ":::foo", which the Common Lisp reader would not read without complaint. Are we not biased in this debate due to SLIME having been made for Common Lisp?
Even the Common Lisp reader will barf when encountering symbols that are in an non-existent package, like those passed through the swank protocol from another lisp system. We had to work around that for the MCLIDE client, which is implemented in Common Lisp.
The easiest compromise is to change %CURSOR-MARKER% to +CURSOR- MARKER+ which, if I read the documentation right, the Clojure reader will accept. Doing so, is sweeping a problem under the carpet.
Yes, I agree, insofar as either protocol participant is using a parser (be it the Common Lisp, Clojure, or ELisp reader) that isn't just reading a string.
However, to recommend that the protocol participants implement a parser from scratch to read these Lisp forms seems unreasonable too; after all, the protocol looks to have been designed for writing and reading Lisp forms directly. It can't just be an accident that there are paired parentheses and keyword-style tags.
A problem is that the swank protocol allows the full common Lisp syntax, with a few self-imposed limitations to make the output compatible with emacs lisp. The consequence is that it is a substantial job to implement a custom reader.
I still think that it is Clojure that should "change". It should provide a way to READ in a form with minimal processing -- something that CL's reader, for instance, does not really allow to unfortunate consequences.
Does using the protocol require a parser separate from a lowest-common-denominator Lisp reader? If so, would you introduce syntax that the Common Lisp reader couldn't accept?
Now that there are multiple clients and servers, I think it is time that we formalize the lisp syntax of the messages in the swank protocol. It should be simple, limited to a subset of what is allowed by Common Lisp... something parseable with say less than a page of lisp code. Swank clients and servers should adhere to this syntax in the messages they produce, but should not be required to implement a validating parser for reading. Implementations may continue to use the lisp reader as before, although writing a custom reader would be trivial with a well defined and simple message syntax. -- Terje Norderhaug terje@in-progress.com
* Terje Norderhaug [2010-01-09 22:52+0100] writes:
Now that there are multiple clients and servers, I think it is time that we formalize the lisp syntax of the messages in the swank protocol. It should be simple, limited to a subset of what is allowed by Common Lisp... something parseable with say less than a page of lisp code.
Here's a one page reader: (defun simple-read () (let ((c (read-char))) (case c (#\" (with-output-to-string (*standard-output*) (loop for c = (read-char) do (case c (#\" (return)) (#\\ (write-char (read-char))) (t (write-char c)))))) (#\( (loop collect (simple-read) while (ecase (read-char) (#\) nil) (#\space t)))) (#\' `(quote ,(simple-read))) (t (let ((string (with-output-to-string (*standard-output*) (loop for ch = c then (read-char nil nil) do (case ch ((nil) (return)) ((#\space #\)) (unread-char ch) (return)) (t (write-char ch))))))) (cond ((digit-char-p (aref string 0)) (parse-integer string)) ((intern string)))))))) Now prove or disprove that Slime uses this syntax. Helmut
Helmut Eller <heller@common-lisp.net> writes:
* Terje Norderhaug [2010-01-09 22:52+0100] writes:
Now that there are multiple clients and servers, I think it is time that we formalize the lisp syntax of the messages in the swank protocol. It should be simple, limited to a subset of what is allowed by Common Lisp... something parseable with say less than a page of lisp code.
Here's a one page reader:
(defun simple-read () (let ((c (read-char))) (case c (#\" (with-output-to-string (*standard-output*) (loop for c = (read-char) do (case c (#\" (return)) (#\\ (write-char (read-char))) (t (write-char c)))))) (#\( (loop collect (simple-read) while (ecase (read-char) (#\) nil) (#\space t)))) (#\' `(quote ,(simple-read))) (t (let ((string (with-output-to-string (*standard-output*) (loop for ch = c then (read-char nil nil) do (case ch ((nil) (return)) ((#\space #\)) (unread-char ch) (return)) (t (write-char ch))))))) (cond ((digit-char-p (aref string 0)) (parse-integer string)) ((intern string))))))))
Now prove or disprove that Slime uses this syntax.
The slime-fuzzy contrib uses floats over the wire. :-) -T.
"Tobias C. Rittweiler" <tcr@freebits.de> writes:
Helmut Eller <heller@common-lisp.net> writes:
* Terje Norderhaug [2010-01-09 22:52+0100] writes:
Now that there are multiple clients and servers, I think it is time that we formalize the lisp syntax of the messages in the swank protocol. It should be simple, limited to a subset of what is allowed by Common Lisp... something parseable with say less than a page of lisp code.
Here's a one page reader:
(defun simple-read () (let ((c (read-char))) (case c (#\" (with-output-to-string (*standard-output*) (loop for c = (read-char) do (case c (#\" (return)) (#\\ (write-char (read-char))) (t (write-char c)))))) (#\( (loop collect (simple-read) while (ecase (read-char) (#\) nil) (#\space t)))) (#\' `(quote ,(simple-read))) (t (let ((string (with-output-to-string (*standard-output*) (loop for ch = c then (read-char nil nil) do (case ch ((nil) (return)) ((#\space #\)) (unread-char ch) (return)) (t (write-char ch))))))) (cond ((digit-char-p (aref string 0)) (parse-integer string)) ((intern string))))))))
Now prove or disprove that Slime uses this syntax.
The slime-fuzzy contrib uses floats over the wire. :-)
-T.
Hmm, actually no, slime-fuzzy turns them into strings. I misremebered, it's C-c C-c/C-c C-k that sends floats over the wire as part of a compilation result. -T.
A main reason for swank messages to encode some symbols as strings has been differences in how lisps escape characters in symbols. I suggest we formalize the symbol syntax in such a way that we no longer have to encode symbols as strings in swank messages. Just agreeing on using a backslash will do. On Jan 10, 2010, at 12:15 AM, Helmut Eller wrote:
* Terje Norderhaug [2010-01-09 22:52+0100] writes:
Now that there are multiple clients and servers, I think it is time that we formalize the lisp syntax of the messages in the swank protocol. It should be simple, limited to a subset of what is allowed by Common Lisp... something parseable with say less than a page of lisp code.
Here's a one page reader:
(defun simple-read () (let ((c (read-char))) (case c (#\" (with-output-to-string (*standard-output*) (loop for c = (read-char) do (case c (#\" (return)) (#\\ (write-char (read-char))) (t (write-char c)))))) (#\( (loop collect (simple-read) while (ecase (read-char) (#\) nil) (#\space t)))) (#\' `(quote ,(simple-read))) (t (let ((string (with-output-to-string (*standard-output*) (loop for ch = c then (read-char nil nil) do (case ch ((nil) (return)) ((#\space #\)) (unread-char ch) (return)) (t (write-char ch))))))) (cond ((digit-char-p (aref string 0)) (parse-integer string)) ((intern string))))))))
-- Terje Norderhaug terje@in-progress.com
On Jan 10, 2010, at 12:15 AM, Helmut Eller wrote:
* Terje Norderhaug [2010-01-09 22:52+0100] writes:
Now that there are multiple clients and servers, I think it is time that we formalize the lisp syntax of the messages in the swank protocol. It should be simple, limited to a subset of what is allowed by Common Lisp... something parseable with say less than a page of lisp code.
Here's a one page reader:
(defun simple-read () (let ((c (read-char))) (case c (#\" (with-output-to-string (*standard-output*) (loop for c = (read-char) do (case c (#\" (return)) (#\\ (write-char (read-char))) (t (write-char c)))))) (#\( (loop collect (simple-read) while (ecase (read-char) (#\) nil) (#\space t)))) (#\' `(quote ,(simple-read))) (t (let ((string (with-output-to-string (*standard-output*) (loop for ch = c then (read-char nil nil) do (case ch ((nil) (return)) ((#\space #\)) (unread-char ch) (return)) (t (write-char ch))))))) (cond ((digit-char-p (aref string 0)) (parse-integer string)) ((intern string))))))))
Now prove or disprove that Slime uses this syntax.
Shouldn't we allow the empty list "()" ? -- Terje Norderhaug terje@in-progress.com
On Jan 14, 2010, at 01:11 , Terje Norderhaug wrote:
(cond ((digit-char-p (aref string 0)) (parse-integer string)) ((intern string))))))))
Now prove or disprove that Slime uses this syntax.
Shouldn't we allow the empty list "()" ?
Or the (very useful) symbol '0+ ... I understand SIMPLE-READ mostly to be a straw-man, FWIW. When you're finished plugging all the holes it will be BIG-HAIRY-READ... ;) -- Cheers, Michael
On Jan 14, 2010, at 12:35 AM, Michael Weber wrote:
On Jan 14, 2010, at 01:11 , Terje Norderhaug wrote:
Shouldn't we allow the empty list "()" ?
Or the (very useful) symbol '0+ ...
I understand SIMPLE-READ mostly to be a straw-man, FWIW. When you're finished plugging all the holes it will be BIG-HAIRY-READ... ;)
No, I think it already is fine as definition of the simple syntax to use for swank messages passed between the client and server. We should not make it big and hairy in an attempt to cover all the variety in Lisp. I still think the protocol should provide a way to escape characters in symbols so we can use them freely, but that's pretty much it: (defun read-swank () (let ((c (read-char))) (case c (#\" (with-output-to-string (*standard-output*) (loop for c = (read-char) do (case c (#\" (return)) (#\\ (write-char (read-char))) (t (write-char c)))))) (#\( (loop collect (simple-read) while (ecase (read-char) (#\) nil) (#\space t)))) (#\' `(quote ,(simple-read))) (t (let ((string (with-output-to-string (*standard-output*) (loop for ch = c then (read-char nil nil) do (case ch ((nil) (return)) (#\\ (write-char (read-char))) ((#\space #\)) (unread-char ch) (return)) (t (write-char ch))))))) (cond ((digit-char-p c) (parse-integer string)) ((intern string)))))))) -- Terje Norderhaug terje@in-progress.com
On Nov 17, 2009, at 12:46 PM, Tobias C. Rittweiler wrote:
At the moment, the Clojure backend does not correctly implement parsing the protocol.
Last time I looked into the Clojure implementation of swank, it apparently rely on a number of expectations that easily might be broken by a swank client, such as that essential symbols are passed in lowercase. Maybe they could be helped by a more rigidly defined protocol, like a strict syntax for swank symbols that the Clojure backend developers and others could use to ensure compatibility. -- Terje Norderhaug
On Nov 17, 2009, at 12:46 PM, Tobias C. Rittweiler wrote:
"Stefan Kamphausen" <skampi@gmx.net> writes:
Hi Tobias,
-------- Original-Nachricht --------
Datum: Tue, 17 Nov 2009 21:04:21 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
"Stefan Kamphausen" <skampi@gmx.net> writes: [...]
You didn't answer the important question:
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
I can change %cursor-marker% to have a symbol name that's nicer for clojure to cope with. But it would be an interim solution, only. I really want to see this fixed in Clojure.
Changing to slime::%cursor-marker% to, say. slime::-cursor-marker- fixes the problem. I know it may not be the ideal solution, but it would be nice to have a version of slime newer than the one from last october that worked with clojure. thanks, Cyrus
Cyrus Harmon <ch-slime@bobobeach.com> writes:
On Nov 17, 2009, at 12:46 PM, Tobias C. Rittweiler wrote:
"Stefan Kamphausen" <skampi@gmx.net> writes:
Hi Tobias,
-------- Original-Nachricht --------
Datum: Tue, 17 Nov 2009 21:04:21 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
"Stefan Kamphausen" <skampi@gmx.net> writes: [...] You didn't answer the important question:
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names? I can change %cursor-marker% to have a symbol name that's nicer for clojure to cope with. But it would be an interim solution, only. I really want to see this fixed in Clojure.
Changing to slime::%cursor-marker% to, say. slime::-cursor-marker- fixes the problem. I know it may not be the ideal solution, but it would be nice to have a version of slime newer than the one from last october that worked with clojure.
Clojure's swank implementation has to replicate SIMPLE-READ in swank-rpc.lisp. It's only two dozens lines of code, should not be hard. -T.
Am 21.05.2010 um 09:40 schrieb Tobias C. Rittweiler:
Cyrus Harmon <ch-slime@bobobeach.com> writes:
On Nov 17, 2009, at 12:46 PM, Tobias C. Rittweiler wrote:
"Stefan Kamphausen" <skampi@gmx.net> writes:
Hi Tobias,
-------- Original-Nachricht --------
Datum: Tue, 17 Nov 2009 21:04:21 +0100 Von: "Tobias C. Rittweiler" <tcr@freebits.de> An: slime-devel@common-lisp.net Betreff: Re: [slime-devel] broken clojure REPL
"Stefan Kamphausen" <skampi@gmx.net> writes: [...] You didn't answer the important question:
Clojure seems to conflate how symbols are named, and how they're read in. Is there no way to escape symbol names?
Just to answer this question, in case it hasn't been answered: AFAICT there is no way to escape symbol names, in particular you can't have "%". From "http://clojure.org/reader": "Symbols begin with a non-numeric character and can contain alphanumeric characters and *, +, !, -, _, and ? (other characters will be allowed eventually, but not all macro characters have been determined). '/' has special meaning, it can be used once in the middle of a symbol to separate the namespace from the name, e.g. my-namespace/foo. '/' by itself names the division function. '.' has special meaning - it can be used one or more times in the middle of a symbol to designate a fully-qualified class name, e.g. java.util.BitSet, or in namespace names. Symbols beginning or ending with '.' are reserved by Clojure. Symbols containing / or . are said to be 'qualified'. Symbols beginning or ending with ':' are reserved by Clojure. A symbol can contain one or more non-repeating ':'s."
I can change %cursor-marker% to have a symbol name that's nicer for clojure to cope with. But it would be an interim solution, only. I really want to see this fixed in Clojure.
Changing to slime::%cursor-marker% to, say. slime::-cursor-marker- fixes the problem. I know it may not be the ideal solution, but it would be nice to have a version of slime newer than the one from last october that worked with clojure.
Clojure's swank implementation has to replicate SIMPLE-READ in swank-rpc.lisp. It's only two dozens lines of code, should not be hard.
Having said the above you still might be able to manually handle the %cursor-marker% symbol and change it to something else in Clojure swank. But then again I don't know anything about swank / clojure / ... Best regards, Niko
Nikolaus Demmel <demmeln@in.tum.de> writes:
Clojure's swank implementation has to replicate SIMPLE-READ in swank-rpc.lisp. It's only two dozens lines of code, should not be hard.
Having said the above you still might be able to manually handle the %cursor-marker% symbol and change it to something else in Clojure swank. But then again I don't know anything about swank / clojure /
SIMPLE-READ does not involve CL:READ. It's a simplistic clone of it for the purposes of the slime sexp format. Clojure's swank implementation should similiarly _not_ use the reader that comes with Clojure. -T.
On May 21, 2010, at 3:43 AM, Tobias C. Rittweiler wrote:
Nikolaus Demmel <demmeln@in.tum.de> writes:
Clojure's swank implementation has to replicate SIMPLE-READ in swank-rpc.lisp. It's only two dozens lines of code, should not be hard.
Having said the above you still might be able to manually handle the %cursor-marker% symbol and change it to something else in Clojure swank. But then again I don't know anything about swank / clojure /
SIMPLE-READ does not involve CL:READ. It's a simplistic clone of it for the purposes of the slime sexp format. Clojure's swank implementation should similiarly _not_ use the reader that comes with Clojure.
I have already written a clojure replica of swank-rpc.lisp. It is included in the new version 1.2 of swank for clojure, which was released earlier this week: http://github.com/terjenorderhaug/swank-clojure/commit/ 054613d756884218b984c4418c617379b9bf8fcc -- Terje Norderhaug
I also use clojure with slime from CVS, and it would be interesting to have swank-clojure in CVS version Stefan Kamphausen at "Sun, 15 Nov 2009 13:32:27 +0100" wrote: ....
I ask myself, how many Clojure users are reading this list and are using Slime from CVS? Maybe enough to consider integrating official support for Clojure?
-- With best wishes, Alex Ott, MBA http://alexott.blogspot.com/ http://xtalk.msk.su/~ott/ http://alexott-ru.blogspot.com/
participants (9)
-
Alex Ott -
Cyrus Harmon -
Helmut Eller -
Michael Weber -
Nikolaus Demmel -
Stefan Kamphausen -
Steven E. Harris -
Terje Norderhaug -
Tobias C. Rittweiler