Hi!
Generating a core with pre-loaded packages (namely sb-bsd-sockets, sb-posix, sb-introspect, sb-cltl2, asdf) is a common optimization technique for faster REPL startups. The technique is detailed in the SLIME and SLY manuals.
Some may argue that modern machines are fast enough. But from personal experience even on very fast hardware, I find that it makes for a much smoother experience, especially for people using SLIME / SLY as their shell :)
I've implemented my own Emacs helper (https://github.com/joaotavora/sly/pull/366), but in https://bugs.launchpad.net/sbcl/+bug/1904042 it was suggested to bring the issue to ASDF which might be the most appropriate place where to share a helper for everyone, independently of the Lisp implementation or editor.
So what about adding a function the would dump an optimized core a bit like I did in the SLY pull request:
--8<---------------cut here---------------start------------->8--- (defvar *default-core-extra-modules* '(asdf))
(defun asdf:dump-core (&key (path *asdf-default-core-location*) (extra-modules *default-core-extra-modules*)) ... ;; Complete the extra-modules with implementation specific modules, as documented in the SLIME / SLY manuals. (cond ((sbcl-p) (append '(sb-bsd-sockets sb-posix sb-introspect sb-cltl2) extra-modules)) ((ccl-p) ... --8<---------------cut here---------------end--------------->8---
Then all the user would have to do is:
--8<---------------cut here---------------start------------->8--- CL-USER> (asdf:dump-core) --8<---------------cut here---------------end--------------->8---
And restart their REPL, assuming they've configured their REPL to use this new core.
It can also be useful to only replace an existing core if the modules or the Lisp version are different. This would allow the user to systematically run `asdf:dump-core` before starting their REPL, thus automatically ensuring they run an optimized core.
--8<---------------cut here---------------start------------->8--- ;; In Emacs: (setq sly-lisp-implementations '((sbcl (lambda () (call-process "sbcl" nil t nil "--no-userinit" "--eval" "(require :asdf)" "--eval" "(asdf:core-dump)" "--quit") `(("sbcl" "--core" path-to-core)))))) --8<---------------cut here---------------end--------------->8---
Thoughts?
ASDF already has a build-image command, and I believe you could do what you want by creating a "shell" system that depends on the packages you want (sb-bsd-sockets, sb-posix, sb-introspect, sb-cltl2), and build an image for that system. See `image-op` in the ASDF manual. If that doesn't work for you, please post again to explain why it won't, and we can try to improve things.
Best, R
On 17 Nov 2020, at 4:20, Pierre Neidhardt wrote:
Hi!
Generating a core with pre-loaded packages (namely sb-bsd-sockets, sb-posix, sb-introspect, sb-cltl2, asdf) is a common optimization technique for faster REPL startups. The technique is detailed in the SLIME and SLY manuals.
Some may argue that modern machines are fast enough. But from personal experience even on very fast hardware, I find that it makes for a much smoother experience, especially for people using SLIME / SLY as their shell :)
I've implemented my own Emacs helper (https://github.com/joaotavora/sly/pull/366), but in https://bugs.launchpad.net/sbcl/+bug/1904042 it was suggested to bring the issue to ASDF which might be the most appropriate place where to share a helper for everyone, independently of the Lisp implementation or editor.
So what about adding a function the would dump an optimized core a bit like I did in the SLY pull request:
--8<---------------cut here---------------start------------->8--- (defvar *default-core-extra-modules* '(asdf))
(defun asdf:dump-core (&key (path *asdf-default-core-location*) (extra-modules *default-core-extra-modules*)) ... ;; Complete the extra-modules with implementation specific modules, as documented in the SLIME / SLY manuals. (cond ((sbcl-p) (append '(sb-bsd-sockets sb-posix sb-introspect sb-cltl2) extra-modules)) ((ccl-p) ... --8<---------------cut here---------------end--------------->8---
Then all the user would have to do is:
--8<---------------cut here---------------start------------->8--- CL-USER> (asdf:dump-core) --8<---------------cut here---------------end--------------->8---
And restart their REPL, assuming they've configured their REPL to use this new core.
It can also be useful to only replace an existing core if the modules or the Lisp version are different. This would allow the user to systematically run `asdf:dump-core` before starting their REPL, thus automatically ensuring they run an optimized core.
--8<---------------cut here---------------start------------->8--- ;; In Emacs: (setq sly-lisp-implementations '((sbcl (lambda () (call-process "sbcl" nil t nil "--no-userinit" "--eval" "(require :asdf)" "--eval" "(asdf:core-dump)" "--quit") `(("sbcl" "--core" path-to-core)))))) --8<---------------cut here---------------end--------------->8---
Thoughts?
-- Pierre Neidhardt https://ambrevar.xyz/
Ha, that's a good starting point indeed!
So I made the following system:
--8<---------------cut here---------------start------------->8--- (defsystem "ambrevar/dump" :class :package-inferred-system :depends-on #.(append '(ambrevar/all asdf) #+sbcl '(sb-bsd-sockets sb-posix sb-introspect sb-cltl2)) :build-operation image-op :build-pathname #+sbcl "sbcl.core-for-sly" #+ccl "ccl.core-for-sly") --8<---------------cut here---------------end--------------->8---
And I'm building it with:
--8<---------------cut here---------------start------------->8--- sbcl --no-userinit --eval '(require :asdf)' --eval '(asdf:make :ambrevar/dump)' --eval '(quit)'
## And to test portability... ccl --no-init --eval '(require :asdf)' --eval '(asdf:make :ambrevar/dump)' --eval '(quit)' --8<---------------cut here---------------end--------------->8---
What's missing is a way to rebuild the image when the compiler changes (e.g. SBCL gets updated).
One way would be to load the core, return the (lisp-implementation-version), compare against the current compiler (lisp-implementation-version). But that's too much code.
Alternatively, I can just try to start the implementation with the given core. If the version is wrong, it will fail. This is less code, but still (untested):
--8<---------------cut here---------------start------------->8--- (defun ambrevar/dump-lisp-core (implementation core-flag extra-flags) (cl-flet ((find-core () (first (directory-files ambrevar/lisp-core-root nil (format "%s.*" implementation))))) (let ((core (find-core))) (when (or (not core) ;; Not right version (/= 0 (call-process implementation nil nil nil core-flag core extra-flags "--eval" "(quit)"))) (let ((lisp-output-buffer (get-buffer-create " *Lisp dump log*"))) (with-current-buffer lisp-output-buffer (erase-buffer)) (apply #'call-process implementation nil (list lisp-output-buffer t) nil "--eval" "(require :asdf)" "--eval" "(asdf:make :ambrevar/dump)" "--eval" "(quit)")))) (or core (find-core)))) --8<---------------cut here---------------end--------------->8---
Would there be a way to tell ASDF "dump image only if version differs"?
I'm not sure this is doable with ASDF, because this means starting an implementation and loading ASDF, which is already too slow.
So we need a way to tell that the image is out of date in a fraction of a second (<50ms I'd say), i.e. without loading ASDF.
Maybe create a new (contrib?) system that loads in an instant with just one function, one that checks if a core is valid or not.
Thoughts?
-- Pierre Neidhardt https://ambrevar.xyz/
What I would recommend is to set up a fake component type for the host system, and then the system definition will automatically detect a change.
The difficulty would be having the host system version be recorded in the filesystem so that ASDF can see it.
I'd suggest you have an operation that reads a file with a version number in it, and updates that file if the SBCL version number has increased. Then if ASDF sees that the file's date is new, it will know that SBCL has been updated and rebuild everything.
But... that requires comparing dates on input and output files, so you might need a *pair* of such files that you rotate -- you read the input file, and then write an output file, and ASDF will compare the dates on the two and decide whether it needs to update.
That should be enough to get you started.
On 17 Nov 2020, at 11:47, Pierre Neidhardt wrote:
Ha, that's a good starting point indeed!
So I made the following system:
--8<---------------cut here---------------start------------->8--- (defsystem "ambrevar/dump" :class :package-inferred-system :depends-on #.(append '(ambrevar/all asdf) #+sbcl '(sb-bsd-sockets sb-posix sb-introspect sb-cltl2)) :build-operation image-op :build-pathname #+sbcl "sbcl.core-for-sly" #+ccl "ccl.core-for-sly") --8<---------------cut here---------------end--------------->8---
And I'm building it with:
--8<---------------cut here---------------start------------->8--- sbcl --no-userinit --eval '(require :asdf)' --eval '(asdf:make :ambrevar/dump)' --eval '(quit)'
## And to test portability... ccl --no-init --eval '(require :asdf)' --eval '(asdf:make :ambrevar/dump)' --eval '(quit)' --8<---------------cut here---------------end--------------->8---
What's missing is a way to rebuild the image when the compiler changes (e.g. SBCL gets updated).
One way would be to load the core, return the (lisp-implementation-version), compare against the current compiler (lisp-implementation-version). But that's too much code.
Alternatively, I can just try to start the implementation with the given core. If the version is wrong, it will fail. This is less code, but still (untested):
--8<---------------cut here---------------start------------->8--- (defun ambrevar/dump-lisp-core (implementation core-flag extra-flags) (cl-flet ((find-core () (first (directory-files ambrevar/lisp-core-root nil (format "%s.*" implementation))))) (let ((core (find-core))) (when (or (not core) ;; Not right version (/= 0 (call-process implementation nil nil nil core-flag core extra-flags "--eval" "(quit)"))) (let ((lisp-output-buffer (get-buffer-create " *Lisp dump log*"))) (with-current-buffer lisp-output-buffer (erase-buffer)) (apply #'call-process implementation nil (list lisp-output-buffer t) nil "--eval" "(require :asdf)" "--eval" "(asdf:make :ambrevar/dump)" "--eval" "(quit)")))) (or core (find-core)))) --8<---------------cut here---------------end--------------->8---
Would there be a way to tell ASDF "dump image only if version differs"?
I'm not sure this is doable with ASDF, because this means starting an implementation and loading ASDF, which is already too slow.
So we need a way to tell that the image is out of date in a fraction of a second (<50ms I'd say), i.e. without loading ASDF.
Maybe create a new (contrib?) system that loads in an instant with just one function, one that checks if a core is valid or not.
Thoughts?
-- Pierre Neidhardt https://ambrevar.xyz/
Hi Robert,
Thanks again for the feedback.
"Robert Goldman" rpgoldman@sift.info writes:
What I would recommend is to set up a fake component type for the host system, and then the system definition will automatically detect a change.
Sorry, I don't understand what you mean with this, can you elaborate a little bit?
The difficulty would be having the host system version be recorded in the filesystem so that ASDF can see it.
I'd suggest you have an operation that reads a file with a version number in it, and updates that file if the SBCL version number has increased. Then if ASDF sees that the file's date is new, it will know that SBCL has been updated and rebuild everything.
But... that requires comparing dates on input and output files, so you might need a *pair* of such files that you rotate -- you read the input file, and then write an output file, and ASDF will compare the dates on the two and decide whether it needs to update.
If I understand you correctly, in the first link I've sent there is an Emacs snippet that does that:
https://github.com/joaotavora/sly/pull/366/files
It works, but the point is that ideally all users should not have to roll their own trick: it would be greaet if ASDF (because it's the common denominator of all Lisp implementations) would provide a helper that does this job, thus saving some hassle to the users.
Does that make sense? Interested in your thoughts.
Cheers!
I came to the conclusion that I needed a wrapper script.
The answer is here:
https://gitlab.com/ambrevar/lisp-repl-core-dumper
It's a small tool to easily generate fast-loading core, optionally with your favourite systems preloaded:
lisp-repl-core-dumper sbcl alexandria fset
The above generates an image (only if needed) then loads it and drops you in the REPL.
Use the above command to set the entry point to a REPL such as SLIME or SLY, it makes for near instant startup time! This is particularly useful if you use a Common Lisp REPL as your shell and fire up new REPL instances all the time!
Even if you think that the REPL starts fast enough and that you don't need to restart your image much, this wrapper still makes for a more comfortable experience since it's dead simple to use, close to zero setup.
Bonuses:
- Compiler version changes are detected and the image will be automatically regenerated.
- Multiple images can be used simultaneously, thus you can easily switch from an image with just Alexandria to an image with just Fset.
Feedback welcome!
-
Pierre Neidhardt -
Robert Goldman