Hi,
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
Regards,
Ala'a Mohammad.
This is the best thing I've read on the Pro list so far, I think.
I'm a filer, but as I deploy multiple cloud instances, I'm starting to think about the advantages of imaging.
However, I have little experience with it. Especially on Debian.
Anybody have good resources on imaging that are not obtainable through a superficial web search?
On Thu, Jan 20, 2011 at 7:45 AM, Ala'a Mohammad amalawi@gmail.com wrote:
Hi,
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
Regards,
Ala'a Mohammad.
pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro
On 20 January 2011 11:16, Jacob Kozinn jacobkozinn@gmail.com wrote:
This is the best thing I've read on the Pro list so far, I think. I'm a filer, but as I deploy multiple cloud instances, I'm starting to think about the advantages of imaging. However, I have little experience with it. Especially on Debian. Anybody have good resources on imaging that are not obtainable through a superficial web search?
For making images, there is * Xach's buildapp, which is great but sbcl-only * my own cl-launch, which is portable, and can be complemented with my command-line-argument library, or Didier Verna's CLON (ported only to a few platforms).
Both rely on ASDF2, which any modern implementation should provide, and/or that you can get by yourself.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Lie, n.: A very poor substitute for the truth, but the only one discovered to date.
I haven't looked at cl-launch in a while, but remember it looking promising. Thanks.
I also looked briefly into ASDF2 but can't remember recognizing any advantages to moving to it from ASDF. I am sure the advantages are there though. What am I missing?
On Thu, Jan 20, 2011 at 8:34 AM, Faré fahree@gmail.com wrote:
On 20 January 2011 11:16, Jacob Kozinn jacobkozinn@gmail.com wrote:
This is the best thing I've read on the Pro list so far, I think. I'm a filer, but as I deploy multiple cloud instances, I'm starting to
think
about the advantages of imaging. However, I have little experience with it. Especially on Debian. Anybody have good resources on imaging that are not obtainable through a superficial web search?
For making images, there is
- Xach's buildapp, which is great but sbcl-only
- my own cl-launch, which is portable, and can be complemented with my
command-line-argument library, or Didier Verna's CLON (ported only to a few platforms).
Both rely on ASDF2, which any modern implementation should provide, and/or that you can get by yourself.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Lie, n.: A very poor substitute for the truth, but the only one discovered to date.
Faré fahree@gmail.com wrote:
For making images, there is
- Xach's buildapp, which is great but sbcl-only
- my own cl-launch, which is portable, and can be complemented with my
command-line-argument library, or Didier Verna's CLON (ported only to a few platforms).
More precisely, Clon supports SBCL, CMU-CL, CCL, ECL, CLISP, and ABCL (upcoming release) on Unix (inc. MacOS X). Coming next is support for ACL and LispWorks, but I'll have to beg for fully functionnal versions in order to do the porting.
Clon's job is command-line options, that's all. There is however a couple of things to ease the delivery of standalone applications in the form of the DUMP macro plus some Makefile rules and scripts in the demo/ directory. Everything is documented.
On Thu, Jan 20, 2011 at 4:45 PM, Ala'a Mohammad amalawi@gmail.com wrote:
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
Some reasons why one would use images rather than loading up from files:
- Saving on startup times
Nowadays, using images to reduce startup time of the Lisp is often not worth the trouble. Computers are fast, file systems are fast and most importantly, restarts are relatively rare with most Lisp systems that I know. There certainly are cases where every second in terms of saved startup time is important, and in these cases, writing an image is a good way to go.
- Reducing the number of files needed to distribute an application, or distribute binary applications
Some Lisps can dump images that are self contained. Most of them require some auxiliary files. If you want to distribute your system without providing source code, a dumped image is the easiest way to achive that.
- Save transient state across Lisp invocations
As an image includes the full heap state, one can use saved images to snapshot transient state. That way, one can get around all serialization issues that database systems try to solve. The approach is limited in that the commit units are generally rather large.
Personally, I used saved images rarely. I deploy my systems with source code, so I need to deploy a lot of files anyway, and I use a database system, so I don't need images as a persistence mechanism. Startup times don't matter much to me, as I'm usually working with long-running Lisp instances.
-Hans
Ala'a Mohammad wrote on Thu, Jan 20, 2011 at 07:45:29PM +0400:
Hi,
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
I'm an imager. Even my dotfiles are "converted" into an image. When I log into a new machine I run a script to set up my dotfiles and the Lisp part of it loads my personal libraries, changes settings the way I like and then saves an image.
The major reason is that it's much faster to load the result. I also make it shut up for good so that starting this image doesn't have any output except a (customized) prompt.
It also safeguards you against somebody messing with the OS-installed Lisp.
Martin
[Once again I forgot to Reply All. Ala, you've seen part of this before.]
I'm a filer. With ASDF and even more so these days with Quicklisp, it's just easier to load the stuff I need and it doesn't take that long and it saves me having to spend any mental energy keeping track of different images.
However, Fare's mention of buildapp reminded me that I use it for some of my deployed websites, mostly just so I have one thing that is built and I know won't change, if I accidentally decide to upgrade a library or tweak some source code for some other reason. (Obviously, I could also just keep my source for the running app in some distinct place--now that I'm a bit more adept at git, I might start doing that.)
-Peter
On Thu, Jan 20, 2011 at 7:45 AM, Ala'a Mohammad amalawi@gmail.com wrote:
Hi,
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
Regards,
Ala'a Mohammad.
pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro
On Thursday 20 January 2011 11:49 PM, Peter Seibel wrote:
I'm a filer. With ASDF and even more so these days with Quicklisp, it's just easier to load the stuff I need and it doesn't take that long and it saves me having to spend any mental energy keeping track of different images.
I agree. I have tried using images, but the savings in startup time have not been worth the hassle, atleast not during development.
However, Fare's mention of buildapp reminded me that I use it for some of my deployed websites, mostly just so I have one thing that is built and I know won't change, if I accidentally decide to upgrade a library or tweak some source code for some other reason. (Obviously, I could also just keep my source for the running app in some distinct place--now that I'm a bit more adept at git, I might start doing that.)
Interesting you should mention that. At a previous company, we started with using only images for deployment. Soon, though, we started running into problems with this approach -- during early days of development atleast, it was very painful since we were always patching our live system every now and then. And image files were huge compared to the size of our source code, so transferring them was never fun.
Eventually, what we settled on was to put the source code on our production server(s), but build an image whenever a new version was checked out -- which would ensure quick startup if ever the server was rebooted or the application went down.
A nifty thing which went along with this was a system of patches which could be applied to running production images without restarting them. When we had to push a bug fix to production, we could have done this:
- update the source in the maintenance branch, - update source on production server(s), - shutdown, rebuild, and boot our Lisp system again,
However, shutting down a running Lisp image was not very desirable, so, thanks to the nature of Common Lisp, we managed to go with this instead:
- upload a patch (a .lisp source file) to a designated directory on the server - `load` all the patches in the running Lisp system from said directory (we would ensure that each patch wouldn't break things if load-ed more than once) - ensure that these patches are loaded when the application is started (to account for restarts)
(I hope I've got everything right, this was all over two years ago)
All in all, a combination of images and files worked out fairly well for our production system.
Chaitanya
I use files and quicklisp when developing, but deliver and provision binary images.
Most of my apps are web based, and i run the lisp image behind runit, a service manager that handles logging as well as startup and shutdown, and restarting the image if it crashes. It also makes rolling back a release trivial, modulo database schema changes.
I don't use anything like cl-launch or buildapp, if only because these tools didn't exist in a usable form when i started, and i haven't had a good reason to change.
Quicklisp, OTOH, has changed the way i work significantly. With the exception of production instances, most of my code is deployed via a load.lisp script that explicitly lists and loads dependencies from QL, where before libraries were kept in version control. I'm also using more libraries these days.
So, i'm both, but primarily a filer these days.
Cheers,
drewc
On 20 January 2011 07:45, Ala'a Mohammad amalawi@gmail.com wrote:
Hi,
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
Regards,
Ala'a Mohammad.
pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro
I'd say I'm 70% imager, 30% filer. I usually save an image with libraries I'm using, but then load more stuff on top of that when I start.
-- Scott
On Thu, Jan 20, 2011 at 07:45, Ala'a Mohammad amalawi@gmail.com wrote:
I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
For loading required software, I'd say 100% filer. ASDF and Quicklisp make operating on files to easy for me that I just don't want to waste any energy thinking about what definitions are in the image and which I've saved in a (source-controlled) file somewhere - the 30 seconds I spend waiting for software to load are nothing compared to an afternoon spent debugging a function whose definition in the image differs from what is in the file containing its (alleged) source code... Or worse, functions/variables that are present only in the image, and not in any file.
When it comes to throw-away code, though, I usually just type it into the repl or the slime-scratch buffer. But: once throwaway stuff starts becoming useful, I use quickproject and paste the latest definitions into a source file and check that into git ASAP. Of course, this step has the potential to go wrong in the same way I mentioned above: if I manage to forget a definition, it's debug time again. Fortunately, that happens early enough in the source's lifetime that it's not hard to find the definition again.
As for deploying, I have started making quicklisp trees containing the requisite software and a git checkout of my project. You can still incrementally load stuff, and you can be pretty certain that what you tested truly is what you deployed (-:
Hope that's useful,
It causes me less heartburn during development being a Filer as I use several foreign libraries. But I deploy executables.
-Luke
(Sorry I haven't yet had time to read other responses: am on the road and buried under both 6 inches of Ohio snow and an end month deadline. So apologies if I'm restating other people.)
I'm both.
Development uses images which are primed to check that the system is up to date (and to correct that as appropriate) on startup. That way we get rapid startup but everything compiled up to date.
Occasionally it doesn't look too good (say, an annoying number of GF signatures changed since the image was saved) and the effort of hand-holding the start-time system reload outweighs the effort of rebuilding. Then I quit and rebuild. Mostly when I rebuild I don't force the compile, but if (say) I've been messing with a pervasive macro I'll go all the way.
We've recently introduced the change that force compilation does a force compile of the whole system twice. This allows us to generate warnings on incorrect calls to functions defined later in the system. We aim for the compilation to be warning-free.
The product we ship can't do the above (LispWorks "delivered" image: royalty free but no compile-file) but then we wouldn't want to. The applications come with a patch loader (startup check for fasls in a particular directory).
In case this is a relevant data point: we're developing three applications, 70k lines of code, with a shared library of about 15k lines.
- nick
Hello Nick,
On Fri, Jan 21, 2011 at 10:03 AM, Nick Levine ndl@ravenbrook.com wrote:
Development uses images which are primed to check that the system is up to date (and to correct that as appropriate) on startup. That way we get rapid startup but everything compiled up to date.
Can you describe in more detail how that works?
Cheers,
From: =?ISO-8859-1?Q?Lu=EDs_Oliveira?= luismbo@gmail.com Date: Fri, 21 Jan 2011 10:53:51 +0000
Hello Nick,
On Fri, Jan 21, 2011 at 10:03 AM, Nick Levine ndl@ravenbrook.com wrote:
Development uses images which are primed to check that the system is up to date (and to correct that as appropriate) on startup. That way we get rapid startup but everything compiled up to date.
Can you describe in more detail how that works?
Sure. The detail is a bit mucky and probably unilluminating, because I postpone the compilation until after the LW GUI had finished firing up. (If I weren't so picky, I could hand a restart-function to save-image.) Here are the excerpts from our build script which drive this:
(defun application-load (&key compile-only) (let ((defsys (truename (relative-path "code/defsys.lisp")))) (load defsys)) (compile-system "PROFILER-PLUS" :load (not compile-only) :force (find "-force" sys:*line-arguments-list* :test 'string=)))
(application-load)
[...]
(define-action "Initialize LispWorks Tools" "Reload PPlus" (lambda (screen) (declare (ignore screen)) (when-let (listener (mp:find-process-from-name "Listener 1")) (mp:process-interrupt listener (lambda () (application-load) ;; Now activate the app. (pp::activate nil) )))) :after "Run the environment start up functions")
LispWorks has these beasts called "action lists": hooks by another name. What the above is saying is: while initializing the LW tools, after running LW's internal startup functions, execute this bit of code which as you see recompiles the system (maybe forcing it, depending on the -force line argument) and then activates the application so I can play with it as I hack.
- nick
On Fri, Jan 21, 2011 at 11:37 AM, Nick Levine ndl@ravenbrook.com wrote:
(defun application-load (&key compile-only) (let ((defsys (truename (relative-path "code/defsys.lisp")))) (load defsys)) (compile-system "PROFILER-PLUS" :load (not compile-only) :force (find "-force" sys:*line-arguments-list* :test 'string=)))
I'm not familiar with the details of LW's defsystem. How is this different from loading the application from fasls, speed-wise?
On Jan 21, 2011, at 12:06 PM, Luís Oliveira wrote:
I'm not familiar with the details of LW's defsystem. How is this different from loading the application from fasls, speed-wise?
I'm sure Martin Simmons of LispWorks will correct me if I'm mistaken here, but LW's defsystem can be used to load lisp code, and/or compile and load lisp code, and/or load precompiled fasls, so it spans the whole range.
warmest regards,
Ralph
Raffael Cavallaro raffaelcavallaro@me.com
On Fri, Jan 21, 2011 at 7:50 PM, Raffael Cavallaro raffaelcavallaro@mac.com wrote:
I'm sure Martin Simmons of LispWorks will correct me if I'm mistaken here, but LW's defsystem can be used to load lisp code, and/or compile and load lisp code, and/or load precompiled fasls, so it spans the whole range.
My question is: if you're going to load all of the application's FASLs, what difference does it make that you loaded a dumped image with some previous version of the application? What am I missing?
My question is: if you're going to load all of the application's FASLs,
You're not. load-system (or compile-system with :load t) will do the minimum it has to and leave the rest alone.
what difference does it make that you loaded a dumped image with some previous version of the application? What am I missing?
- n
On Fri, Jan 21, 2011 at 8:56 PM, Luís Oliveira luismbo@gmail.com wrote:
My question is: if you're going to load all of the application's FASLs, what difference does it make that you loaded a dumped image with some previous version of the application? What am I missing?
What I was missing is that when you dump an image, defsystem information about the files' modification times is kept, thus only modified files (and their dependents) will be loaded. *facepalm*
Cheers,
The system Ioad facility I have developed, on its first run on a particular system, creates a core file containing all the support libraries needed for the system, and then starts SBCL using that core file, proceeding to ASDF load the system proper. On subsequent starts, the first core creation step is skipped, saving much time, in most cases making the systems start instantly since no compilation needs to occur.
Whenever site-lisp is changed, then the cores need to be removed and will automatically be recreated when the system restarts. In the core is stored the head git SHA commit id taken from site-lisp at the time of core creation in order to detect a core to source code discrepancy when the cores are ran.
On Thu, Jan 20, 2011 at 4:45 PM, Ala'a Mohammad amalawi@gmail.com wrote:
Hi,
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
Regards,
Ala'a Mohammad.
pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro
For production, I build images, as some of the production instances don't have either NFS or the SBCL build chain.
But for development, I am almost always in slime, being a filer.
Additionally, the production environment does not lend itself to working with a running image.
On Sun, Jan 23, 2011 at 4:51 AM, William Halliburton whalliburton@gmail.com wrote:
The system Ioad facility I have developed, on its first run on a particular system, creates a core file containing all the support libraries needed for the system, and then starts SBCL using that core file, proceeding to ASDF load the system proper. On subsequent starts, the first core creation step is skipped, saving much time, in most cases making the systems start instantly since no compilation needs to occur. Whenever site-lisp is changed, then the cores need to be removed and will automatically be recreated when the system restarts. In the core is stored the head git SHA commit id taken from site-lisp at the time of core creation in order to detect a core to source code discrepancy when the cores are ran.
On Thu, Jan 20, 2011 at 4:45 PM, Ala'a Mohammad amalawi@gmail.com wrote:
Hi,
I'm continually learning Common-lisp and trying to find the best style that suites me better. I've tried 'an imager' style (cooking a an image with all required libraries loaded when required), and 'a filer' style (loading files or systems each time I fire-up a CL implementation). I'm interested to hear what others use CL. How do they manage day to day work? how do their preferred style mesh into their production pipeline (coding, debugging, deployment and maintenance)? and what makes them prefer one way over another or the mix if applicable?
Regards,
Ala'a Mohammad.
pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro
pro mailing list pro@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/pro
lately i'm 99% imager both for development and deployment. most of the time i start from an image that contains all the dependencies we have except the hu.dwim.* stuff and swank.
i don't initiate a reload when the image is started. if i need to do nontrivial work on a dependency, then i just start slime with a vanilla SBCL.
startup time does matter for me because:
- we have 80+ dependencies, a full recompile takes long minutes even on new hardware
- it's easy to lose track of what's the state of the heap and i've debugged ghosts often enough to just simply restart and reload as a first step whenever there's something that baffles me.
- CL has complex load-time/compile-time semantics, and some components (e.g. ASDF) are quite naive regarding this which doesn't help. i have often experienced that what i've done to the heap using redefinitions cannot be loaded from files out of the box. long sessions without a reload mean longer cleanups afterwards when loading from files fail.
relevant sources (http://dwim.hu/darcsweb/darcsweb.cgi):
- hu.dwim.build - the CL build app, similar to Xach's buildapp. it can build itself into an executable.
- hu.dwim.environment - emacs/ for the way emacs/slime is set up; etc/server-scripts for the posix services of production images
production.lisp files throughout our projects. they add something to the way the production images are started.
-
Ala'a Mohammad -
Andreas Fuchs -
Attila Lendvai -
Chaitanya Gupta -
Didier Verna -
Drew Crampsie -
Faré -
Hans Hübner -
Jacob Kozinn -
Luke Crook -
Luís Oliveira -
Martin Cracauer -
Nick Levine -
Peter Seibel -
Raffael Cavallaro -
Scott L. Burson -
William Halliburton -
William Lederer