Re: [slime] swank-loader could use ASDF (#65)
I'm only worried about slow startup time. E.g. (require :asdf) takes about 3 seconds in CCL. if you have ever used ABCL you know that slow startups are no fun.
My ultimate use case for this feature is this one in particular. emacs -Q (require 'package) ;;; unnecessary since emacs 24.1 M-x install-package slime-foo RET ;;; third party slime-foo contrib depends on third party ;;; slime-bar that depends on slime-repl that depends on slime. They ;;; all get installed, in the appropriate ~/.emacs.d/elpa. ;;; ;;; or using the el-get package manager M-x el-get slime-foo RET ;;; same thing happens but they all end up in ~/.emacs.d/el-get Now they all the .el files are spread across across the correct places, having been compiled by the respective package manager. Here is a mock tree of what the file system looks like now: . |-- slime-3.0/ | |-- slime-autoloads.el | |-- slime.el | |-- slime.elc | |-- swank.asd | |-- asdf.lisp | `-- swank.lisp |-- slime-bar-0.5/ | |-- slime-bar-autoloads.el | |-- slime-bar.asd | |-- slime-bar.el | |-- slime-bar.elc | `-- slime-bar.lisp |-- slime-foo-0.1/ | |-- slime-foo-autoloads.el | |-- slime-foo.asd | |-- slime-foo.el | |-- slime-foo.elc | `-- slime-foo.lisp `-- slime-repl-3.0/ |-- slime-repl-autoloads.el |-- slime-repl.el |-- slime-repl.elc |-- slime-repl.asd `-- slime-repl.lisp When the user "M-x slime" the three contribs are setup and loaded. The supporting lisp files are compiled into fasls and stored somewhere appropriate. If a user "M-x slime-connect"s to some running lisp where a swank server is already running, we must attempt to compile any needed supporting lisp there as well. (this might be problematic and we could just assume that the target has the same contribs). Now I think this could be done with or without ASDF. But I think ASDF would make it much simpler. According to Francois, startup times are seem OK. M-x slime on sbcl takes well more than two seconds for me. But * One could see more numbers across more versions: * It's true that for a first time M-x slime on a new lisp implementation, compiling asdf.lisp into a fasl is going to take some time. In those cases I think M-x should show "Compiling ASDF, this only happens once".
The advantage for swank-loader seems marginal to me.
I think with well crafted ASDF systems (and who better than François) this file could well dissapear.
A slightly different idea: assume we have a Makefile and a required "make install" step. During that install step we assume that quicklisp exists or simply download it. If we have quicklisp we have everything we could wish for.
I don't understand, Quicklisp takes at least as long to load as ASDF, since it bundles it. I must be missing something. Also, in the ELPA use-case, the user never "make"s anything. We could package quicklisp, but I think that's going well beyond the scope of slime.
FTR, i have these two patches in my slime for years and haven't experienced any issues (although i never use slime without ASDF): extend the asdf contrib with an asdf system definition search function that finds swank, so that (asdf:load-op :swank) or dependence on it works as expected. https://github.com/attila-lendvai/slime/commit/9ae8a00b2df95d01b0fa3f60f5ada... ;; If ASDF is available then store Slime's fasl's where ASDF stores them. https://github.com/attila-lendvai/slime/commit/f95a4b6f0d0b412ac5eb87cf5b3e4... -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “During times of universal deceit, telling the truth becomes a revolutionary act.” — George Orwell (1903–1950)
On Mon, Jan 06 2014, João Távora wrote:
I'm only worried about slow startup time. E.g. (require :asdf) takes about 3 seconds in CCL. if you have ever used ABCL you know that slow startups are no fun.
My ultimate use case for this feature is this one in particular.
emacs -Q (require 'package) ;;; unnecessary since emacs 24.1 M-x install-package slime-foo RET [...]
My use case is more like: git clone .../slime.git make install
According to Francois, startup times are seem OK. M-x slime on sbcl takes well more than two seconds for me. But
* One could see more numbers across more versions:
* It's true that for a first time M-x slime on a new lisp implementation, compiling asdf.lisp into a fasl is going to take some time. In those cases I think M-x should show "Compiling ASDF, this only happens once".
I'm not worried about the one time cost of compiling lisp files. I'm talking about the cost of loading additional fasl files every time.
The advantage for swank-loader seems marginal to me.
I think with well crafted ASDF systems (and who better than François) this file could well dissapear.
So you say that's a clear advantage? Well, I still clear out ASDF's fasl cache regularly because I don't trust ASDF. And I rather debug the code in swank-loader.lisp than in asdf.lisp.
A slightly different idea: assume we have a Makefile and a required "make install" step. During that install step we assume that quicklisp exists or simply download it. If we have quicklisp we have everything we could wish for.
I don't understand, Quicklisp takes at least as long to load as ASDF, since it bundles it. I must be missing something. Also, in the ELPA use-case, the user never "make"s anything. We could package quicklisp, but I think that's going well beyond the scope of slime.
I would say ELPA packages must be installed and the user "makes" .elc files. Assume we know how to load quicklisp. Let's say the user needs to create a config file, ~/.slime/config, that looks like so: (:load-quicklisp-form (cl-user:load-quicklisp) :load-asdf-form (require :asdf) :slime-contribs (slime-repl slime-foo)) With that we know how to load quicklisp, but that doesn't mean we load it all the time. We would read the config file every time, but load quicklisp, and possibly ASDF, only when we need to install/compile our stuff. Helmut
Helmut Eller <eller.helmut@gmail.com> writes:
My use case is more like:
git clone .../slime.git make install
Sure, that can be served too.
I'm not worried about the one time cost of compiling lisp files. I'm talking about the cost of loading additional fasl files every time.
Yes, I understood that. I was presenting an additional concern regarding compile-times. I'm waiting for François's numbers on the relative proportion of loading ASDF to loading the current swank-loader.lisp across some implementations. That way we can check if by requiring ASDF to be loaded, we're introducing some extraordinary burden on users of slime using these implementations.
So you say that's a clear advantage? Well, I still clear out ASDF's fasl cache regularly because I don't trust ASDF. And I rather debug the code in swank-loader.lisp than in asdf.lisp.
Yes, it seems like an advantage. Well I understand your point about debugging a familiar swank-loader.lisp, it's not so for others (i.e. co-workers not managing to get a contrib loaded or understanding why swank error'ed out and quit their Allegro suddently). ASDF offers good error messages and restarts, which are already familiar if you're in the quicklisp/asdf ecosystem. In general, offloading some responsibilities to a software package that specilize in them is a good policy in my opinion. In my view `slime-init-command' should keep "returning a string to initialize lisp", as per its docstrings. But that string, by customizing a variable: * should, be default, be built around applying ASDF:REQUIRE-SYSTEM to :SWANK and/or whatever contribs the user has set up. * could be based on loading swank-loader.lisp from the current directory, which is the current case. * could be based on some custom swank-loading-logic if the user decides so. Maybe we coulnd move the current swank-loader.lisp to swank-asdfless-loader.lisp and load it as per a `defcustom' that you enable in your config? Still, I'm curious as to why you don't you trust ASDF? Any specific problems in mind?
I would say ELPA packages must be installed and the user "makes" .elc files.
I don't undertand this bit. ELPA packages are uploaded by the developers onto remote repositories and fetched-and-installed via package.el's `package-install' command. They can/should be built by us using a make target that picks out some files from the source repo, auto-generates others like slime-pkg.el. package-install then autogenerates a slime-autoloads.el file and byte compiles the remaining ones, sets up load paths and all that.
Assume we know how to load quicklisp. Let's say the user needs to create a config file, ~/.slime/config, that looks like so:
(:load-quicklisp-form (cl-user:load-quicklisp) :load-asdf-form (require :asdf) :slime-contribs (slime-repl slime-foo))
At first sight, this would personally bug me: why not do this in the user's .emacs in elisp? Why this file? very few EMACS packages have their own config files (I can think of Gnus only) and they are all emacs-lisp, so that they can be perfectly replaced with elisp code in user's init files. I would see no advantage in adding one such file to SLIME. Also, if I understand correctly, this file would make ELPA (and el-get) packaging much more complicated and non-standard. To be clear. I'm not adding a ASDF dependency to slime so that we get ASDF in the lisp runtime, although that is a side effect, unless we go to some lengths to delete the packages afterwards. I'm adding it because it's becoming a standard and is specilized in representing dependencies between systems, which is something we can definitely use to simplify out loading behaviour: * to make addition of third-party contribs easier; * to ease bundling our own existing contribs in separate packages; * to have "one" way of loading swank, less 300 lines to maintain, and one less option-hungry SWANK-LOADER:INIT to worry about. Once we have a simple enough loading behaviour, it's much easier to add installation behaviours that serve: * your "make install" use case * the hacker's clone-my-own case * ELPA's package-install <some-package-depending-on-slime> * el-get
we know how to load quicklisp but doesn't mean we load it all the time.
I'm still confused. When don't we load it? BTW I think quicklisp always loads ASDF.
On Tue, Jan 07 2014, João Távora wrote: [...]
So you say that's a clear advantage? Well, I still clear out ASDF's fasl cache regularly because I don't trust ASDF. And I rather debug the code in swank-loader.lisp than in asdf.lisp.
Yes, it seems like an advantage. Well I understand your point about debugging a familiar swank-loader.lisp, it's not so for others (i.e. co-workers not managing to get a contrib loaded or understanding why swank error'ed out and quit their Allegro suddently). ASDF offers good error messages and restarts, which are already familiar if you're in the quicklisp/asdf ecosystem.
In general, offloading some responsibilities to a software package that specilize in them is a good policy in my opinion.
Offloading stuff to ASDF doesn't sound much different from offloading stuff to quicklisp.
In my view `slime-init-command' should keep "returning a string to initialize lisp", as per its docstrings. But that string, by customizing a variable:
* should, be default, be built around applying ASDF:REQUIRE-SYSTEM to :SWANK and/or whatever contribs the user has set up.
* could be based on loading swank-loader.lisp from the current directory, which is the current case.
* could be based on some custom swank-loading-logic if the user decides so.
Maybe we coulnd move the current swank-loader.lisp to swank-asdfless-loader.lisp and load it as per a `defcustom' that you enable in your config?
Well, we don't need a default, one option for me, and a third option for the exotic cases. Just put be in the exotic case.
Still, I'm curious as to why you don't you trust ASDF? Any specific problems in mind?
As I said in a another message, I had experiences like this: 1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked. Or just a few days ago: This is SBCL 1.1.12-1.fc20, an implementation of ANSI Common Lisp. ... * (require :asdf) * (uiop:with-temporary-file (:stream s) s) debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread #<THREAD "main thread" RUNNING {AC17981}>: Couldn't remove "/tmp/tmp9MFH86" while closing #<SB-SYS:FD-STREAM for "file /tmp/tmp9MFH86" {BAD9B51}>: No such file or directory Or the "upgrade" stuff and Faré telling us how it fails in CMUCL and CCL. None of this inspires trust. But I recognize that the masses have spoken and ASDF is the de-facto standard.
I would say ELPA packages must be installed and the user "makes" .elc files.
I don't undertand this bit. ELPA packages are uploaded by the developers onto remote repositories and fetched-and-installed via package.el's `package-install' command.
They can/should be built by us using a make target that picks out some files from the source repo, auto-generates others like slime-pkg.el. package-install then autogenerates a slime-autoloads.el file and byte compiles the remaining ones, sets up load paths and all that.
All I'm saying is that M-x package-install is not that different from "make install".
Assume we know how to load quicklisp. Let's say the user needs to create a config file, ~/.slime/config, that looks like so:
(:load-quicklisp-form (cl-user:load-quicklisp) :load-asdf-form (require :asdf) :slime-contribs (slime-repl slime-foo))
At first sight, this would personally bug me: why not do this in the user's .emacs in elisp? Why this file?
Because reading .emacs from Common Lisp is difficult. It's also much easier to automatically update a config file that contains only "declarative" stuff and no general Lisp code. And why would you put information about quicklisp into your .emacs in the first place? I could understand .sbclrc or .swank.lisp.
very few EMACS packages have their own config files (I can think of Gnus only) and they are all emacs-lisp, so that they can be perfectly replaced with elisp code in user's init files. I would see no advantage in adding one such file to SLIME.
The requirements of most Emacs packages are quite different. Few Emacs packages ship substantial non-Emacs-Lisp source code around and have to initialize half a dozen different interpreters each with different command line arguments etc. Also, storing per-project settings in a config file in the projects directory seems natural to me. A single .emacs just doesn't work well for that.
Also, if I understand correctly, this file would make ELPA (and el-get) packaging much more complicated and non-standard.
Not sure why that would be.
To be clear. I'm not adding a ASDF dependency to slime so that we get ASDF in the lisp runtime, although that is a side effect, unless we go to some lengths to delete the packages afterwards.
I'm adding it because it's becoming a standard and is specilized in representing dependencies between systems, which is something we can definitely use to simplify out loading behaviour:
* to make addition of third-party contribs easier;
* to ease bundling our own existing contribs in separate packages;
* to have "one" way of loading swank, less 300 lines to maintain, and one less option-hungry SWANK-LOADER:INIT to worry about.
Once we have a simple enough loading behaviour, it's much easier to add installation behaviours that serve:
* your "make install" use case * the hacker's clone-my-own case * ELPA's package-install <some-package-depending-on-slime> * el-get
we know how to load quicklisp but doesn't mean we load it all the time.
I'm still confused. When don't we load it? BTW I think quicklisp always loads ASDF.
We could load quicklisp during "make install". That's the time when you potentially need to download third party Common Lisp code. E.g. when a contrib depends on cl-pcre. We could also use it when a user wants to add a library to his project, e.g. we could have something like M-x slime-quickload. Helmut
On Tue, Jan 7, 2014 at 3:58 PM, Helmut Eller <eller.helmut@gmail.com> wrote:
Offloading stuff to ASDF doesn't sound much different from offloading stuff to quicklisp.
Even less different when quicklisp offloads the system loading work to ASDF. What quicklisp does that ASDF doesn't is download stuff of the network; for compiling and loading the downloaded systems, it delegates to ASDF.
Well, we don't need a default, one option for me, and a third option for the exotic cases. Just put be in the exotic case.
I agree that the endgame should be a single way to load the code that works well for everyone on all platforms.
Still, I'm curious as to why you don't you trust ASDF? Any specific problems in mind?
As I said in a another message, I had experiences like this:
1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked.
Are these experiences recent? If so, that's a troubling bug indeed, either in ASDF or in the system foo. ASDF should not have this kind of behavior anymore, unless system foo is doing something both funky and buggy. If you experience this again, please take it to the asdf-devel mailing-list, it's a matter that we're serious about eliminating.
Or just a few days ago:
This is SBCL 1.1.12-1.fc20, an implementation of ANSI Common Lisp. ... * (require :asdf) * (uiop:with-temporary-file (:stream s) s)
debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread #<THREAD "main thread" RUNNING {AC17981}>: Couldn't remove "/tmp/tmp9MFH86" while closing #<SB-SYS:FD-STREAM for "file /tmp/tmp9MFH86" {BAD9B51}>: No such file or directory
I did a lot of work recently in with-temporary-file, to make it more portable than it was, whereby many use cases would only run on some platforms. I also improved its configurability and defaults, and added proper tests to the test suite, so it should be and remain robust. Unhappily, this means that with-temporary-file isn't portably reliable until the imminent ASDF release 3.1.1. My apologies for the inconvenience.
Or the "upgrade" stuff and Faré telling us how it fails in CMUCL and CCL. None of this inspires trust.
CMUCL has a bad bug in PCL and no one to fix it. Maybe I should think harder about a workaround. Three systems out of almost a thousand in Quicklisp have been found to trigger a weird CLOS bug on CCL after an ASDF upgrade, unless some classes are finalized somehow. We're investigating, but I don't think that's a blocking issue. Upgrades shouldn't be required in most cases, but when they're needed, they're sorely needed (e.g. you need a fix to with-temporary-file, and your implementation doesn't have a recent one). Happily, both CMUCL and CCL keep their provided version of ASDF fairly up to date, so no upgrade should be needed on either platform.
But I recognize that the masses have spoken and ASDF is the de-facto standard.
ASDF has been a de facto standard for a long time now; I've tried to make it actually robust and portable; where I have failed, I'm eager to fix it.
Because reading .emacs from Common Lisp is difficult. It's also much easier to automatically update a config file that contains only "declarative" stuff and no general Lisp code. And why would you put information about quicklisp into your .emacs in the first place? I could understand .sbclrc or .swank.lisp.
I agree that Emacs can pass any necessary information to CL at initialization time based on its configuration, much more easily than CL can be taught (and maintained!) to read Emacs configuration.
The requirements of most Emacs packages are quite different. Few Emacs packages ship substantial non-Emacs-Lisp source code around and have to initialize half a dozen different interpreters each with different command line arguments etc.
I maintain lisp-invocation.asd to handle the same issue for XCVB, but it's not maintainably sharable with Emacs. (Though it could be, as long as maintained as part of SLIME.)
I'm still confused. When don't we load it? BTW I think quicklisp always loads ASDF.
We could load quicklisp during "make install". That's the time when you potentially need to download third party Common Lisp code. E.g. when a contrib depends on cl-pcre. We could also use it when a user wants to add a library to his project, e.g. we could have something like M-x slime-quickload.
That's probably a good idea. Something similar for clbuild would be nice, too. In a production environment, this would be slightly harder, because you'd want to import things into your project's version control. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "Don't worry about people stealing your ideas. If your ideas are any good, you'll have to ram them down people's throats." — Howard Aiken
On Wed, Jan 08 2014, Faré wrote: [...]
Still, I'm curious as to why you don't you trust ASDF? Any specific problems in mind?
As I said in a another message, I had experiences like this:
1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked.
Are these experiences recent?
Maybe not. But I still clean the cache whenever there's the slightest suspicion that ASDF could be related to a problem. I'm sure you can fix bugs once identified, but fixing trust is another issue. Helmut
On Wed, Jan 8, 2014 at 2:53 AM, Helmut Eller <eller.helmut@gmail.com> wrote:
On Wed, Jan 08 2014, Faré wrote:
As I said in a another message, I had experiences like this:
1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked.
Are these experiences recent?
Maybe not. But I still clean the cache whenever there's the slightest suspicion that ASDF could be related to a problem. I'm sure you can fix bugs once identified, but fixing trust is another issue.
Actually, before ASDF2 (mid 2010), no one could "fix bugs once identified": there was no single person in charge, no way to push bug fixes, a lot of bugs, and a good reason not to trust ASDF. But the situation changed completely 3-4 years ago when I took over, made asdf upgradable so that bug fixes could be pushed, and started fixing the bugs and pushing the fixes. ASDF is a completely different beast these days, and well tested thanks to Robert Goldman, Anton Vodonosov, Dave Cooper and more. If your bad experiences were with versions of ASDF older than that, I invite you to reconsider this trust issue. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Whatever you do will be insignificant, but it is very important that you do it. — Gandhi
Helmut Eller <eller.helmut@gmail.com> writes:
In general, offloading some responsibilities to a software package that specilize in them is a good policy in my opinion. Offloading stuff to ASDF doesn't sound much different from offloading stuff to quicklisp.
True. But still ASDF is common on many implementations systems while Quicklisp isn't (yet) and skeptic coworkers, for instance, offer much less resistance to the former. But I'm also not necessarily opposed to recommending, bundling, or installing quicklisp. The latter too seem a little out of SLIME's jurisdiction though.
Well, we don't need a default, one option for me, and a third option for the exotic cases. Just put be in the exotic case.
Did you mean "put *me*" :) ? Anyway I've noticed the :init argument to `slime-start' and the :init keyword in `slime-lisp-implementations' is more than sufficient abstraction to provide the two options rather cleanly. Even to disallow ASDF loading in problematic implementations.
1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked.
FWIW, though they are rare, I've recently found similar cache clearing problems in vanilla slime as well: M-. would bring me to a correctly named but empty file in some /tmp/slime-$PID dir. Clearing the slime fasl cache did the trick. I don't know how to reproduce though.
Or the "upgrade" stuff and Faré telling us how it fails in CMUCL and CCL. None of this inspires trust.
Indeed
But I recognize that the masses have spoken and ASDF is the de-facto standard.
Indeed++.
All I'm saying is that M-x package-install is not that different from "make install".
Not needing to drop to a shell or running an external command is a big usability difference. Emacs -Q M-x package-install RET slime-foo RET M-x slime Notice no `slime-setup' to setup the foo contrib and its dependencies. That said I'm still *not* saying that the only way to do this is with ASDF and without swank-loader.lisp. Indeed for (legitimate) cases of mistrust, swank-loader.lisp could still be the default. But eventually we should deprecate it.
At first sight, this would personally bug me: why not do this in the user's .emacs in elisp? Why this file? Because reading .emacs from Common Lisp is difficult. It's also much
Ahh, ok I get it finally, You want to read that file from the CL side. That makes more sense. Still I don't like yet another file to configure... I already have ~/.swank and ~/.emacs
Also, storing per-project settings in a config file in the projects directory seems natural to me. A single .emacs just doesn't work well for that.
Your case makes a bit more sense to me now I understand you want to read from CL. But it is yet another file... :-( Anyway recent emacsen have many mechanisms to manage per-project configuration. "directory-local variables" are probably the simplest, but very powerful already [1]. You do need emacs 24 though, but it is already very easy to get in on debian stable. In emacs 23 you have to bake your own, but it's easy too. I must say I might have missed something but still don't understand how per-project settings. Maybe I don't understand which settings you want to set, and how many of them relate to this discussion. But if your use case is avoiding getting ASDF loaded into your image in some projects but not others, a directory-localized `slime-lisp-implementations' and its existing `:init' arg fit the bill rather nicely I think.
Also, if I understand correctly, this file would make ELPA (and el-get) packaging much more complicated and non-standard. Not sure why that would be.
To be fair, though, you're right, *packaging* itself would not be affected. But M-x package-install, IIUC is dedicated to installing and activating a package by extracting autoload definitions for functions. The autoloaded function, once run, slime could create your default ~/.slime.d/config with the correct pathname of the quicklisp.lisp also stored in a autoloaded variable.
I'm still confused. When don't we load it? BTW I think quicklisp always loads ASDF. We could load quicklisp during "make install". That's the time when you potentially need to download third party Common Lisp code. E.g. when a contrib depends on cl-pcre.
I think those contribs could well depend on a slime-quicklisp contrib, docstring being "Ensure quicklisp is available in the host lisp and provide some convenience commands".
We could also use it when a user wants to add a library to his project, e.g. we could have something like M-x slime-quickload.
A good command for that slime-quicklisp contrib, but something I would keep out of core. Anyway to summarize: * in pull request #80 I'm working with Faré on a setup that uses ASDF/swank-loader.lisp per user defcustom. It is still somewhat crudely hacked, but I will iterate and make it cleaner. * In trunk I've added a Makefile that creates elpa packages. They are not yet installable. As explained, it should be possible to allow these freer scheme without requiring ASDF at all. In a future pull request we can discuss this final ASDFless version and then decide if we pull in #80 as well. João [1] http://www.gnu.org/software/emacs/manual/html_node/emacs/Directory-Variables...
On Wed, Jan 08 2014, João Távora wrote:
1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked.
FWIW, though they are rare, I've recently found similar cache clearing problems in vanilla slime as well: M-. would bring me to a correctly named but empty file in some /tmp/slime-$PID dir. Clearing the slime fasl cache did the trick. I don't know how to reproduce though.
That's because the test.sh script copies the source to /tmp/ and removes it later. The last changelog date of the fasl and the source match and the source is older than the fasl; so the cache is, arguably, not out of date. Whether fasl files should contain absolute file names is another matter. Helmut
On Wed, Jan 08 2014, João Távora wrote:
But I'm also not necessarily opposed to recommending, bundling, or installing quicklisp. The latter too seem a little out of SLIME's jurisdiction though.
The problem "How do we get from a vanilla Lisp to one where we can do (require :asdf)" was solved by forcing every implementation to bundle ASDF. IMO, quicklisp is facing a similar problem. Should quicklisp use a similar "bundle it with the implementation" strategy? I think not, because a) it needs a lot of communication b) the bundled version gets outdated quickly. <aside> As far as I understand, the "upgrade" mechanism in ASDF is only needed because the bundled version may be too old. </aside I think this annoying but important problem could be solved more elegantly by agreeing on the name of a configuration file and the names of some configuration variables. If not a configuration file then a .init file, that we all agree to load from .sbclrc or .ccl-init etc. Yes, this problem has not much to do with SLIME or Emacs but nobody else seems solve it or think about it much. Helmut
Helmut Eller <eller.helmut@gmail.com> writes:
On Wed, Jan 08 2014, João Távora wrote:
But I'm also not necessarily opposed to recommending, bundling, or installing quicklisp. The latter too seem a little out of SLIME's jurisdiction though.
The problem "How do we get from a vanilla Lisp to one where we can do (require :asdf)" was solved by forcing every implementation to bundle ASDF. IMO, quicklisp is facing a similar problem. Should quicklisp use a similar "bundle it with the implementation" strategy? I think not, because a) it needs a lot of communication b) the bundled version gets outdated quickly.
For what it's worth, I would prefer implementations do not include Quicklisp. I don't like the idea of implementations making divergent code for Quicklisp, and I don't like the idea of the instructions for how to get Quicklisp diverging based on which Lisp you use. (asdf-install was bad in this regard.) The instructions for Quicklisp are kept intentionally simple as a result: - download http://beta.quicklisp.org/quicklisp.lisp - (load "quicklisp.lisp") Zach
: Helmut The problem "How do we get from a vanilla Lisp to one where we can do (require :asdf)" was solved by forcing every implementation to bundle ASDF. IMO, quicklisp is facing a similar problem. Should quicklisp use a similar "bundle it with the implementation" strategy? I think not, because a) it needs a lot of communication b) the bundled version gets outdated quickly.
I promise I never forced or threatened any Lisp vendor — though I nagged and probably annoyed many of them. It's just every Lisp vendor's interest to boast out of the box support for hundreds of free software libraries. One big different between ASDF and Quicklisp, though, is that ASDF is relatively small, self-contained, and, if not forward-compatible, at least self-upgradable since ASDF2. That means that a vendor can update whenever he likes, and be confident that if ASDF gets obsolete, that's no big deal, because it can be used to bootstrap the loading of the new version in a portable way, automated way (assuming that new version was properly installed by whoever or whatever software needs it) On why upgradability matters in such cases: http://fare.livejournal.com/149264.html Quicklisp is more open-ended (unless you always distribute all the libraries together with it, in which case that's BIG and gets obsolete fast), and so far as I know solves the hard upgrade issues by assuming standardized per-user location, some (minimal) level of user interaction and possible Lisp process restart if any major incompatibility occurs. That makes it a harder proposition to bundle Quicklisp with either an implementation, or, in that case, SLIME; but also less necessary. I still think integration of SLIME with Quicklisp is a good idea, just not co-distribution.
: Zach For what it's worth, I would prefer implementations do not include Quicklisp. I don't like the idea of implementations making divergent code for Quicklisp, and I don't like the idea of the instructions for how to get Quicklisp diverging based on which Lisp you use. (asdf-install was bad in this regard.)
The instructions for Quicklisp are kept intentionally simple as a result:
- download http://beta.quicklisp.org/quicklisp.lisp
- (load "quicklisp.lisp")
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The penalty for laughing in a courtroom is six months in jail; if it were not for this penalty, the jury would never hear the evidence. — H. L. Mencken
Hi all, For what little they're worth, here are my votes on the topic. Q: Have Quicklisp distribute Slime? A: Sure! (Doesn't it already?) Q: Have Slime rely on ASDF? A: Sure. The current Slime loader is generally reliable, but there is something to be said for pooling effort in a common solution, and ASDF has really improved in recent years. Q: Have Slime rely on Quicklisp? A: No way! You've got this all backwards. Quicklisp adds downloading, dependency management, and such to ASDF. It also assumes a network connection and a certain level of trust in the repository. Q: Have Slime rely on make? A: ?!?? GNU Make maybe, but it is essentially a non-starter on MSWin, Android, iOS, .... Thinking about an ideal, non-Quicklisp-assisted install, could something like the following work? - download and extract Slime (zip, tar.gz, or txz) - open slime-install.lisp in Emacs - follow some instructions that start an interactive session that configures and installs (or upgrades) Slime Later, Daniel
Je Wed, 8 Jan 2014 22:58:06 -0500 (EST) Daniel Herring <dherring@tentpost.com> skribis:
Hi all,
For what little they're worth, here are my votes on the topic.
Q: Have Quicklisp distribute Slime? A: Sure! (Doesn't it already?)
Q: Have Slime rely on ASDF? A: Sure. The current Slime loader is generally reliable, but there is something to be said for pooling effort in a common solution, and ASDF has really improved in recent years.
Q: Have Slime rely on Quicklisp? A: No way! You've got this all backwards. Quicklisp adds downloading, dependency management, and such to ASDF. It also assumes a network connection and a certain level of trust in the repository.
Q: Have Slime rely on make? A: ?!?? GNU Make maybe, but it is essentially a non-starter on MSWin, Android, iOS, ....
Thinking about an ideal, non-Quicklisp-assisted install, could something like the following work?
- download and extract Slime (zip, tar.gz, or txz) - open slime-install.lisp in Emacs - follow some instructions that start an interactive session that configures and installs (or upgrades) Slime
Dear List I'd like to put in my two pennorth. I am currently using the wheezy flavour of the Debian Linux distro and I should like to point out that slime is a standard Debian package. Both ASDF and Quicklisp are available as the Debian packages cl-asdf and cl-quicklisp respectively. I also use sbcl and I notice that the ASDF bundled with sbcl is not as recent as that supplied by the Debian package cl-asdf. So, maybe some time should be given as to why that is so. Yours aye -- Dr Sian Mountbatten <http://www.poenikatu.co.uk/> Associate member of the FSF. No. 10888
On Thu, 9 Jan 2014, Sian Mountbatten wrote:
Je Wed, 8 Jan 2014 22:58:06 -0500 (EST) Daniel Herring <dherring@tentpost.com> skribis:
Hi all,
For what little they're worth, here are my votes on the topic.
Q: Have Quicklisp distribute Slime? A: Sure! (Doesn't it already?)
Q: Have Slime rely on ASDF? A: Sure. The current Slime loader is generally reliable, but there is something to be said for pooling effort in a common solution, and ASDF has really improved in recent years.
Q: Have Slime rely on Quicklisp? A: No way! You've got this all backwards. Quicklisp adds downloading, dependency management, and such to ASDF. It also assumes a network connection and a certain level of trust in the repository.
Q: Have Slime rely on make? A: ?!?? GNU Make maybe, but it is essentially a non-starter on MSWin, Android, iOS, ....
Thinking about an ideal, non-Quicklisp-assisted install, could something like the following work?
- download and extract Slime (zip, tar.gz, or txz) - open slime-install.lisp in Emacs - follow some instructions that start an interactive session that configures and installs (or upgrades) Slime
Dear List
I'd like to put in my two pennorth.
I am currently using the wheezy flavour of the Debian Linux distro and I should like to point out that slime is a standard Debian package. Both ASDF and Quicklisp are available as the Debian packages cl-asdf and cl-quicklisp respectively.
I also use sbcl and I notice that the ASDF bundled with sbcl is not as recent as that supplied by the Debian package cl-asdf. So, maybe some time should be given as to why that is so.
Hi Sian, It is inevitable in general that there be this disparity, and I think that ASDF has allowed for that. From my limited understanding, ASDF does some magic to upgrade itself on the fly. The only other option is that the Common Lisp implementions do not use the internal copy and point to the external copy, and the ASDF maintainer has ruled that a Bad Idea. Regards, Faheem
Daniel Herring <dherring@tentpost.com> writes:
Q: Have Quicklisp distribute Slime? A: Sure! (Doesn't it already?) Q: Have Slime rely on ASDF? A: Sure. The current Slime loader is generally reliable, but there is something to be said for pooling effort in a common solution, and ASDF has really improved in recent years. Q: Have Slime rely on Quicklisp? A: No way! You've got this all backwards. Quicklisp adds downloading, dependency management, and such to ASDF. It also assumes a network connection and a certain level of trust in the repository.
Follow pull request #83, which proposes to implement exactly these options.
Thinking about an ideal, non-Quicklisp-assisted install, could something like the following work?
- download and extract Slime (zip, tar.gz, or txz) - open slime-install.lisp in Emacs - follow some instructions that start an interactive session that configures and installs (or upgrades) Slime
It could work, but doesn't have to be that complicated, i.e. that interactive session is too much I think. One of the following emacs-only instructions should suffice: ELPA (require 'package) ;; in emacs 24, or get package.el for 23 M-x package-install RET slime RET ;; OR M-x package-install RET slime-some-contrib RET M-x slime M-x slime-connect ;; .. to some server running a compatible ;; swank version El-get: (require el-get) ;; not in either emacs, but very popular (el-get-install 'slime) ;; OR (el-get-install 'slime-some-contrib) M-x slime ;; OR M-x slime-connect ;; .. to some server running a compatible ;; swank version Clone-my-own or tgz dist: (add-to-list 'load-path "<path-to-your-unpacked-slime>") (add-to-list 'load-path "<path-to-bundled-contribs>") (add-to-list 'load-path "<path-to-some-third-party-contrib>") (require 'slime) ;; OR (require 'slime-autoloads) ;; ...cuts your init time (slime-setup '(my contribs)) M-x slime ;; OR M-x slime-connect ;; .. to some server running a compatible ;; swank version This is my plan, and Helmut seems to agree with it. It will be made easier if we merge #83 and use ASDF as the default slime loading method, but is not impossible otherwise. Note that "slime-some-contrib" might be a bundled contrib like slime-repl or slime-fancy, or some third-party contrib. The quicklisp install can stay put, of course, but as soon as these options are provided, there is no need to launch a lisp, then install elisp and then ask you to change your .emacs init file. But if that's what you prefer, OK.
My use case is more like:
git clone .../slime.git make install
How would that affect (or not) your .emacs? How is emacs supposed to find the correct slime? One still has to add it to the load-path, I presume. Is this supposed to add SLIME to your CL source-registry? Or is Emacs supposed to point CL to the files that correspond to the current SLIME's installation? What when one is using a remote SLIME?
According to Francois, startup times are seem OK. M-x slime on sbcl takes well more than two seconds for me. But
M-x slime on sbcl takes about three seconds for me, but ASDF is a small part of that (I use swank-asdf, so it is part of the slime startup already).
I'm not worried about the one time cost of compiling lisp files. I'm talking about the cost of loading additional fasl files every time.
Incidentally, because I use swank-asdf (and seemingly, it's part of the default contribs on sbcl and ccl), I'm paying the price for it anyway, at which point I may as well enjoy some benefits from it, so it's no "additional fasl" to me. Admittedly, it is a small additional fasl for people who were not previously using that contrib, and/or who explicitly disabled it on sbcl or ccl. (.2s extra load time on CCL.)
I think with well crafted ASDF systems (and who better than François) this file could well dissapear.
So you say that's a clear advantage? Well, I still clear out ASDF's fasl cache regularly because I don't trust ASDF. And I rather debug the code in swank-loader.lisp than in asdf.lisp.
Well, there would still be a top-level file, but swank-loader could be reduced to (require :asdf) (asdf:load-system :swank) (swank:init) or some such, with a more elaborate fallback for implementations that don't provide asdf. Have you had bad experiences with ASDF's fasl cache since ASDF 2.000? I agree that before then, the situation was quite bad, but since then, apart from slightly rough transitions when CCL made its fasl numbering per-target, or when Allegro introduced smp vs non-smp fasls, which I believe swank must have experienced, too, ASDF's fasl cache has always been just as good as swank's (from which it is proudly derived, after all — with many thanks). I see a few small advantages to ASDF's fasl cache: * it is configurable in a way that is important for production code; IIRC, at ITA, we trick $HOME as being under the build tree so that swank would store its fasls in a controlled place. One configuration is better than two. * it matches source by path location, which means more sharing when you upgrade SLIME (recompiling as needed, no more no less) yet no accidental sharing when two subtly different versions at different locations have a same version tag. That said if you've had issues, I'm curious to know what they were, and to make sure they have been addressed in the current (or else next) release of ASDF.
I would say ELPA packages must be installed and the user "makes" .elc files.
Assume we know how to load quicklisp. Let's say the user needs to create a config file, ~/.slime/config, that looks like so:
(:load-quicklisp-form (cl-user:load-quicklisp) :load-asdf-form (require :asdf) :slime-contribs (slime-repl slime-foo))
With that we know how to load quicklisp, but that doesn't mean we load it all the time. We would read the config file every time, but load quicklisp, and possibly ASDF, only when we need to install/compile our stuff.
That mostly makes sense. Beware though that it is better to load ASDF before Quicklisp, and to load an ASDF version no older than the one required by Quicklisp. While ASDF is designed to be upgraded, some implementations seem to have issues when an old ASDF is loaded, then upgraded (CMUCL totally fails upgrade, and CCL looks like it has odd corner cases at times after upgrade). That means it's better to manage the loading of ASDF than to let two independent mechanisms introduce discrepancy and fail. Frankly, I'm the first hater and doubter of ASDF, but at the same time, I favor more declarative and more manageable installation of software, and I believe ASDF helps much more than it costs (ASDF 1 admittedly wasn't quite there yet, and I totally understand why SLIME was built without it; however, ASDF 2.26 is OK, and ASDF 3 is quite decent I believe.) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Mathematics is the Queen of Science but she isn't very Pure; she keeps having babies by handsome young upstarts and various frog princes. — Donald Kingsbury (In "psychohistorical crisis", 2001)
On Tue, Jan 07 2014, Faré wrote:
My use case is more like:
git clone .../slime.git make install
How would that affect (or not) your .emacs? How is emacs supposed to find the correct slime? One still has to add it to the load-path, I presume.
make could ask if it should write or update ~/.slime/config. It could ask if it should update .emacs and add the directory to load-path or perhaps install some ELPA style package with autoloads.
Is this supposed to add SLIME to your CL source-registry?
It could add SWANK. But that doesn't seem necessary.
Or is Emacs supposed to point CL to the files that correspond to the current SLIME's installation?
Shouldn't matter if Lisp and Emacs read the same ~/.slime/config.
What when one is using a remote SLIME?
swank-loader would read the remote ~/.slime/config.
I'm not worried about the one time cost of compiling lisp files. I'm talking about the cost of loading additional fasl files every time.
Incidentally, because I use swank-asdf (and seemingly, it's part of the default contribs on sbcl and ccl), I'm paying the price for it anyway, at which point I may as well enjoy some benefits from it, so it's no "additional fasl" to me. Admittedly, it is a small additional fasl for people who were not previously using that contrib, and/or who explicitly disabled it on sbcl or ccl. (.2s extra load time on CCL.)
Well: Welcome to Clozure Common Lisp Version 1.9-dev-r15611M-trunk (LinuxX8664)! ? (time (require :asdf)) (REQUIRE :ASDF) took 3,118,425 microseconds (3.118425 seconds) to run.
I think with well crafted ASDF systems (and who better than François) this file could well dissapear.
So you say that's a clear advantage? Well, I still clear out ASDF's fasl cache regularly because I don't trust ASDF. And I rather debug the code in swank-loader.lisp than in asdf.lisp.
Well, there would still be a top-level file, but swank-loader could be reduced to (require :asdf) (asdf:load-system :swank) (swank:init) or some such, with a more elaborate fallback for implementations that don't provide asdf.
Have you had bad experiences with ASDF's fasl cache since ASDF 2.000? I agree that before then, the situation was quite bad, but since then, apart from slightly rough transitions when CCL made its fasl numbering per-target, or when Allegro introduced smp vs non-smp fasls, which I believe swank must have experienced, too, ASDF's fasl cache has always been just as good as swank's (from which it is proudly derived, after all — with many thanks).
I see a few small advantages to ASDF's fasl cache: * it is configurable in a way that is important for production code; IIRC, at ITA, we trick $HOME as being under the build tree so that swank would store its fasls in a controlled place. One configuration is better than two. * it matches source by path location, which means more sharing when you upgrade SLIME (recompiling as needed, no more no less) yet no accidental sharing when two subtly different versions at different locations have a same version tag.
That said if you've had issues, I'm curious to know what they were, and to make sure they have been addressed in the current (or else next) release of ASDF.
I don't track ASDF versions. All I know is that a couple of times, I did essentially: 1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked. Helmut
On Mon, Jan 6, 2014 at 8:04 PM, Helmut Eller <eller.helmut@gmail.com> wrote:
How would that affect (or not) your .emacs? How is emacs supposed to find the correct slime? One still has to add it to the load-path, I presume.
make could ask if it should write or update ~/.slime/config. It could ask if it should update .emacs and add the directory to load-path or perhaps install some ELPA style package with autoloads.
Yes. I'm not up-to-date as to latest Emacs package best practices; we should ask someone who knows for sure.
Or is Emacs supposed to point CL to the files that correspond to the current SLIME's installation? Shouldn't matter if Lisp and Emacs read the same ~/.slime/config.
Makes sense. Can the path be overridden? How? For production code, you usually don't want to rely on stuff in ~/
Welcome to Clozure Common Lisp Version 1.9-dev-r15611M-trunk (LinuxX8664)! ? (time (require :asdf)) (REQUIRE :ASDF) took 3,118,425 microseconds (3.118425 seconds) to run.
Yes, I filed a bug for that and that was fixed in 15963. http://trac.clozure.com/ccl/ticket/1111 You may want to update your CCL.
That said if you've had issues, I'm curious to know what they were, and to make sure they have been addressed in the current (or else next) release of ASDF.
I don't track ASDF versions. All I know is that a couple of times, I did essentially:
1. (asdf:load-system foo), 2. then had some load-time problem, 3. cleaned out the cache, 4. and magically (asdf:load-system foo) worked.
That sounds totally probable for ASDF 1 and very early ASDF 2 — actually, this kind of unstability was one of the reasons that made me start hacking ASDF. But that sounds quite surprising in ASDF 2.003 or later (Jun 2010) — unless the problem was failure to recompile things that depend on modified systems, in which case the problem was only completely fixed in ASDF 3 (February 2013). If this memory is more recent than that, this is definitely a bug worth reporting. PS: To João, Allegro ships with ASDF 2.23.7, but if you run update.sh, it will upgrade you to a quite recent 3.0.2.3. Regards, —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Few facts are more revealing than the direction people travel when they vote with their feet. — Don Boudreaux http://bit.ly/afZgx2
participants (8)
-
Attila Lendvai -
Daniel Herring -
Faheem Mitha -
Faré -
Helmut Eller -
joaotavora＠gmail.com -
Sian Mountbatten -
Zach Beane