April 2012 - asdf-devel - mailman3.common-lisp.net

[asdf-devel] asdf 2.21 released
by Faré 29 Apr '12

29 Apr '12

I am pleased to announce the release of ASDF 2.21. Since previous release 2.20, the changes are as follow: * New features: * Most importantly, encodings. ASDF now lets you specify an :encoding for a system, module or component, that is used when loading or compiling Lisp files. See the documentation. By default, the only useful value is :utf-8, and we recommend you use UTF-8 everywhere. We intend to make UTF-8 the default in the future (current default is the legacy behavior of using whichever implicit default your underlying implementation is currently configured to use). An extension asdf-encodings is available that supports more encodings, including autodetection of encoding from either emacs-style declaration -*- Mode: Lisp ; coding: utf-8 -*- or content. * require-system works like load-system, but won't try to load or update systems that have already been loaded, as listed by loaded-systems. More generally, you can specify a list of systems as :force-not argument to load-system and the specified systems won't be loaded or updated. * You can specify :force-not and a list of systems when loading a system, * You can now specify :hostname in your asdf-output-translations, so you can easily share a home directory via the network yet split its fasl cache between several subtly different machines. * Bug Fixes: * lp#982285. since 2.014.4, the default source-registry was skipping anything from /usr or XDG_DATA_DIRS, because of a missing test in getenv-absolute-pathnames. Fixed in .15. * In 2.019.9 aka 2.20, ECL support was broken by using a function in the wrong package (defined in cl-user). Fixed in .1. * logical-pathname support was somewhat broken since 2.017.6, due to the way most implementations fail to read physical namestrings when *default-pathname-defaults* is a logical-pathname. Fixed, together with other logical-pathname issues, and a test case was added to the test suite to ensure no further regression. Works great modulo quirks around implementation bugs on CLISP and Allegro. * Make upgrading ASDF more robust on CMUCL, LispWorks, SBCL. * Minor tweaks: * Use :unspecific in pathname components on more implementations. * export and/or document more utilities. * add a few missing compatfmt for Genera. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org As Marx should have said, the principle of communism is From each according to what he can't get out of doing. To each according to whom he knows. — John McCarthy

1 0

[asdf-devel] ASDlite - light-weight excerpt from ASDF
by Dmitriy Ivanov 26 Apr '12

26 Apr '12

Hello folks, Excuse my bothering if this is not a correct place to announce. Nonetheless... ASDlite Design Goals 1. Small footprint. 2 Ease of embedding into applications and systems not related to "compile-and-load Lisp files" tasks. For example: YstokHelp (http://lisp.ystok.ru/yhelp/) 3. ASDF compatibility for basic needs. 4. Operation arguments specification in dependencies. Observation and Rationale 1. The ASDF built-in operation hierarchy is actually of two-level depth. Original ASDF code does not exploit operation inheritance much (though something can be found in asdf-ecl.lisp). 2. The operation slots are rather useless: original-initargs Is only referred in print-object parent In principle, indicates the target operation that required this one. But due to reusing operation objects in make-sub-operation, this is not the case. Also used for marking along with other visiting slots during traverse but we follow another approach. Adding entirely new operations, i.e. on the first level, is fine. But there is comfortable way to refine existing operations: the easiest way is to add slots to base operation classes as only those properties do get passed into dependency operations. There is a more simple way pass arguments from operate to operation functions - by means of key arguments! 3. The :do-first initarg is actually ignored by ASDF - its always set to ((compile-op (load-op ,@depends-on)))) 4. Avoid inline methods, which are rather inelegant in ASDF: - they rely on eval, - ASDF tries to remove a method from a generic function of a different name. Due to non-standard behavior of remove-method in LW 4.4, system redefinition intrusively signals about this. 5. Referring to features in component definition is more useful than for dependency rules. Operations in ASDlite operation ::= keyword | operation-instance operation-type ::= keyword | type-symbol operation-designator ::= keyword | (keyword . plist) | type-symbol | operation-instance Operations are passed to perform and other operation-specific methods. Operation designators can be used in the right-hand side of rules. We encourage using simple keywords like :compile or :load. For these, ASDlite defines corresponding methods with eql specializers. The plist allows you to pass key arguments to the operation methods. In the "normal" mode, ASDlite accepts only keyword-based forms. If you feel these are not enough and need "full-fledged" ASDF operation classes, you can switch to the ASDF compatibility mode as follows: - push :asdf-compat into *features* before compiling asdlite.lisp, - refer to asdf:compile-op, asdf:load-op and the like, - define your own subclasses of the operation class etc. In the compatibility mode, ASDlite accepts all the above forms of operations and designators. Downloads and Details http://lisp.ystok.ru/asdlite/ -- Sincerely, Dmitriy Ivanov lisp.ystok.ru

2 2

[asdf-devel] asdf-bundle
by Faré 25 Apr '12

25 Apr '12

Dear Juanjo, is it OK with you if unbundle asdf-ecl from asdf.asd, and instead create a somewhat portable asdf extension called asdf-bundle.asd ? Impact: when the unbundling is complete, ecl users will be encouraged to (require :asdf-bundle) instead or in addition to (require :asdf) if they want to use the extensions currently provided in asdf-ecl. But it will make the bundle-op operations available to other implementations in a more portable way. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org To authorize all commercial relations between consenting adults.

2 3

[asdf-devel] Tool for transcoding lisp source files and adding coding file options.
by Douglas Crosher 25 Apr '12

25 Apr '12

A tool to add the coding file option to lisp source files is now available. This can also be used to transcode all the files to UTF-8, or to attempt transcoding to ISO-8859-1 or US-ASCII with loss in comments. There is an example for the current Quicklisp releases with a few required exceptions. This is experimental, particularly the transcoding to ISO-8859-1 or US-ASCII. Please backup your code before giving this a try! See: http://www.scieneer.com/files/transcode.lisp With the coding file option in place, the developmental version of ASDF is able to pass the correct external-format to 'load and 'compile-file. Note the lambda-reader is handled by substituting references to the lambda character back to "(lambda". Since the :encoding file option should become unnecessary with the coding file option, and also because it would be invalid upon any transcoding, it is removed from .asd files. I don't think it would be practical to try and update the :encoding declarations. With the coding file option on all the files, all that asdf-encodings needs to do is read this file option. The UTF-8 and other detection could be removed and moved into a tools such as this. If some of the CL implementations would add the reading of the file option as an external-format then the coding file option could be used by 'load and 'compile-file outside of ASDF - then for these CL implementations asdf-encodings would not be necessary and ASDF could pass on the encoding-file-option reading external-format which may well be the :default. I will look at implementing this for the Scieneer CL, and perhaps take a look at CMUCL, but it would be best optimized for each CL implementation. Regards Douglas Crosher

2 1

Re: [asdf-devel] Faster loading for deployed systems?
by Erik Pearson 24 Apr '12

24 Apr '12

I thought I'd report back briefly on some results. First, I spent some time attempting to implement "perform with session state". This is the idea that when working with ASDF, it would be useful to be able to invoke or create specific behavior for my use case and also to be able to capture work session state without relying on globals. This was partially successful, but I broke something and ran out of patience. ASDF is some wonderful code, but takes a bit to grok it all. So I took the crude approach of just installing a global and using it to accumulate loaded files from within a perform. I created an .asd file which defined the dependencies I wanted the image to have. The loaded it with load-system. Then at the end of load-system I inserted code to save a structure containing a list of the *defined-systems* as well as a list of the loaded files (in order of loading). The next step was to create a little loader script which loads all of the files, and then populates *modules* via satisfies. There were a couple of glitches. For one, a couple of systems had hard coded asdf dependencies. After removing them, they compiled fine. Then another system required asdf-system-connections. I needed to remove that out of the file load list because it isn't necessary after things are loaded. Finally the load list works, the image saves and starts just fine. I can use regular ol' REQUIRE to provide a sanity check, although I realize this may not be the best approach, at least it works without extra code. This saved about 10% on the image size compared to asdf-loading the same systems, and cleared out some unnecessary dependencies. It will take a while to work out the best "batteries included" image. I can always load ASDF for a given project if it needs to incorporate other systems, smaller projects can just load themselves. Of course for development I'll still be using ASDF a hundred times a day! Erik. On Mon, Apr 23, 2012 at 2:10 PM, Robert Goldman <rpgoldman(a)sift.info> wrote: > On 4/23/12 Apr 23 -3:52 PM, Erik Pearson wrote: > > Hi Robert, > > What I was thinking is that OPERATE assembles the plan via TRAVERSE, > > then executes it via PERFORM-PLAN, which calls PERFORM to translate the > > abstraction of operation + component into an action with side effect. It > > seems to me that it is each PERFORM (or, rather, the ones that we are > > interested in) that needs to be captured, since it is only in the body > > of the PERFORM method that the abstractions are made concrete in into > > lisp forms which carry out the desired actions. > > > > What I am referring to as "recording" is to append the lisp forms > > produced by each PERFORM into a list or other structure. At present this > > structure would need to be a global, but ideally it would be a slot in a > > session object that threads through each PERFORM. It is because the > > PERFORMS are carried out serially by PERFORM-PLAN according to the plan > > assembled by TRAVERSE that they can be "played back" to recreate the > > actions of this asdf session. Now if there were parallel executions of > > PERFORMs by PERFORM-PLAN, that would be different... > > > > I hope that makes sense. > > I suppose, but I don't know how you break into what PERFORM does. > A quick and dirty way, as I described above, does require breaking into PERFORM. But I think you could either create an alternate method and inject that at around the PERFORM-PLAN level, or use a SESSION object as an additional specializer argument to open up other possible PERFORM implementations. > > I was thinking you would precompute and cache the plan, and then invoke > PERFORM-PLAN on that. > Or maybe another method altogether to take the place of perform, like SIMULATE-PLAN or RECORD-PLAN, or ASSEMBLE-PLAN. > That assumes that the call to TRAVERSE is actually expensive enough that > this is worth the trouble. I think you could confirm or disconfirm that > claim quickly with some timing experiments. > Yes, I think I did that -- but with a different concern. When I was concerned about ASDF mucking around inspecting files when I was doing a REQUIRE, I added a feature to cause a fully loaded module to be flagged as loaded via mark-operation-done, so that do-traverse would stop traversing at such a module. It was about 2.5 seconds to traverse normally, and about 0.5 with this optimization for a require after the first one (of course for a deployed system not development.) In any case load time is not my only interest. It is also a desire to be able to use ASDF more effectively for more use cases. Erik. cheers, > r >

3 6

[asdf-devel] Please test asdf-encodings
by Faré 23 Apr '12

23 Apr '12

Dear all, asdf-encodings now includes a version of Douglas Crosher's encoding detection algorithm. It is also automatically disabled on implementations without unicode support. As compared to Douglas's version, the detection algorithm uses :ascii or :latin1 instead of :default as a fallback when no declaration was found and no UTF-n encoding was detected. It also uses a 1024-byte buffer rather than 320-byte buffer, to imitate what Emacs does with respect to the beginning of a file. I suggest that asdf-encodings is now ready for testing, and invite you to test it. An example package that uses it is my lambda-reader, heavily modified from Brian Mastenbrook's original to make it hopefully portable to all implementations with or without utf-8 support. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Sin lies only in hurting other people unnecessarily. All other "sins" are invented nonsense. (Hurting yourself is not sinful — just stupid.) — Robert Heinlein, "Time Enough For Love"

2 2

[asdf-devel] Update: encoding file options version.
by Douglas Crosher 23 Apr '12

23 Apr '12

A revised version of ASDF replacing the system definition :encoding support with support for reading the encoding from file options is now available at: http://www.scieneer.com/files/asdf.lisp Changes: * Can read the file options in a wider range of files. Also supports C, Python, XML, and probably others. It's more robust and may be of some utility in other code. * EBCDIC support removed to save on the line count. Can add this back at a cost of about 40 lines. * UTF-8 detection removed to save on the line count. Still detects a UTF-8 BOM, and reads a UTF-8 encoding file option. Adding a UTF-8 encoding file option will help other tools too anyway. * Removed the lengthy external-format translation table to save on the line count. It should be easy for CL implementations to include more aliases so move the burden of maintaining the aliases to the CL implementation. CLISP and the Scieneer CL already support an extensive range of codes and aliases, and an update set of aliases has been sent for CMUCL. SBCL, ECL, and CCL all have a useful set of aliases, but could consider adding a few more. A hook is include for translating the external-formats just in case it is needed, and it is a list of functions to call in turn so multiple projects could add hooks. * Changed the asdf.lisp coding file option to us-ascii from utf-8 - it's a highly portable tool and this should be adequate. * This version is now down to just 114 lines more than the standard version, and the bulk of the code is just one function. * Tested on CCL, CLISP, CMUCL, ECL, SBCL, and Scieneer CL, * Added a large set of test files that exercise the reading of the encoding file option and try to include enough characters from each character set to check that the encoding option has been successful mapped to an appropriate external-format. This includes all the encoding in linux 'iconv -l' that include support for the characters needed by CL, but excludes the EBCDIC codes. All 629 tests pass on the Scieneer CL, 628 on CLISP, and a much smaller but useful sent on CCL, CMUCL, ECL, and SBCL due to their limited aliases and limited encoding support. See: http://www.scieneer.com/files/coding-tests.tar.bz2 Keep in mind that even if a library author chooses an encoding that your CL implementation does not support, if they have added the encoding file option then the files can be reliably and automatically converted to something supported such as UTF-8. There may be a tool out there already in the Emacs or Python community to do this, or something can be written and could run on CLISP which is free and has extensive coding support. Regards Douglas Crosher

4 8

[asdf-devel] Fwd: Faster loading for deployed systems?
by Erik Pearson 22 Apr '12

22 Apr '12

Hi Robert, On Sun, Apr 22, 2012 at 3:24 PM, Erik Pearson <erik(a)defun-web.com> wrote: > Hi Robert, > > Thanks for the Sunday afternoon reply. > > On Sun, Apr 22, 2012 at 2:31 PM, Robert Goldman <rpgoldman(a)sift.info>wrote: > >> On 4/22/12 Apr 22 -3:47 PM, Faré wrote: >> > Dear Erik, >> > >> >> >> ASDF assumes that since the system def file has not changed, the system >> >> hasn't been changed either (I know there has been discussion of this >> >> assumption.) But lets assume that at least part of the reason for that >> is >> >> because it is very time consuming to check the entire set of files >> every >> >> time the developer wants to recompile and load just a single file. >> >> >> > I don't think that's true, but my head hurts just thinking about it >> > Sorry, this was me being confused by my own tracing of asdf operations, and not thinking before I typed. I had placed a debugging line in the call to "safe-file-write-date" to see how ofen this is called in during a load. I thought it was not being called on source files just .asd files, but this was incorrect. I had probably already changed something, or been confused about the multiple calls made on systems -- the first call will get the file write date on both source and compiled (if any), but subsequent calls only on the compiled. Erik.

1 0

[asdf-devel] Fwd: Faster loading for deployed systems?
by Erik Pearson 22 Apr '12

22 Apr '12

Hi Robert, Thanks for the Sunday afternoon reply. On Sun, Apr 22, 2012 at 2:31 PM, Robert Goldman <rpgoldman(a)sift.info> wrote: > On 4/22/12 Apr 22 -3:47 PM, Faré wrote: > > Dear Erik, > > > > >> ASDF assumes that since the system def file has not changed, the system > >> hasn't been changed either (I know there has been discussion of this > >> assumption.) But lets assume that at least part of the reason for that > is > >> because it is very time consuming to check the entire set of files every > >> time the developer wants to recompile and load just a single file. > >> > > I don't think that's true, but my head hurts just thinking about it. > > This is not an accurate understanding. It has come up over and over. > The right thing to do would be to check systems recursively, but there > are a number of reasons why this does not work with the current state of > ASDF. You can look in the archives for explanations; it has been > discussed many times [I am not trying to be rude; I just don't have the > energy to write up another copy of the explanation. Probably we should > put it in as a FAQ in the manual...] > Don't worry. I've been hanging around Lisp long enough to have a high bar on "rude", and this doesn't even come close. Thanks for the correction, I'll go back and revisit those threads. > > > >> Why not assume in a deployment mode that NO system will have changed if > it > >> is already loaded in memory, and unless the application has asked that > it be > >> reloaded? This would be for performance as well as for safety -- we > really > >> don't want dependency checking to result in any filesystem checks, etc. > >> > > Because that's wrong? I don't have as many dependencies as you do when > > I reload a system with asdf, but usually, I do want the dependencies > > to be reloaded. > > The deployment mode is an edge case and can already be handled very > efficiently by dumping an image with the (stable) system you want > already loaded (cf. Xach's buildapp, although I know that's for SBCL, > rather than CCL; and CCL:SAVE-APPLICATION -- > http://ccl.clozure.com/manual/chapter4.9.html#Saving-Applications) In > some sense you are right, ASDF is slow for this use case, but it's > really not the use case ASDF is looking for. > This is what I've been doing for several years now with no problems (none due to asdf, that is!) I'm exploring now a method for deploying small systems which are based on a large-ish compiled, dumped image, but with small to medium sized systems to be loaded and run as batches. Yes, I could just load these files manually, but would at least like to be able to do run-time inquiries of module availability (ala require), which, it seems to me, is the purview of asdf. Also, it is possible that I'd need to add a specific 3rd party lisp package to a deployment without incorporating it into the master runtime image, and integration via asdf also seems natural. BTW the deployment mode, for me, is the purpose of my development, so it is kinda important. But I realize we all sit in a different place in the development process. The main object of my work is a lisp web server that sits under mod_lisp, but I also need to run lisp processes in batch, as well as plenty of lisp ad-hoc analytical work. I have thought that perhaps ASDF could output a deployment structure, which is a set of instructions for constructing the running system. Not well worded, perhaps, but if you capture the actions of asf you should be able to play them back and create a functionaly equivalent image (albeit not necessarily with asdf loaded.) E.g. if you just emit the filenames that are loaded during (perform load-op cl-source-file), you can just place them into a list and load them in sequence using just simple common lisp machinery. Faster than asdf, but still slow compared to loading a big image. > > And ASDF is already too liberal about assuming stability, in a way that > breaks a lot of people's development practice (see my earlier remarks); > the last thing I would want to do is make it even more likely to miss a > change and build an inconsistent system. > > Best, > r > > _______________________________________________ > asdf-devel mailing list > asdf-devel(a)common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel >

1 0

[asdf-devel] Fwd: Faster loading for deployed systems?
by Erik Pearson 22 Apr '12

22 Apr '12

HI Faré, First, thanks for your excellent work on this and other projects. On Sun, Apr 22, 2012 at 1:47 PM, Faré <fahree(a)gmail.com> wrote: > Dear Erik, > > > I am trying to implement a method of using a "batteries included" lisp > image > > (ccl, linux, 64bit) which can be used for running a variety of lisp > tasks in > > a batch mode. I chose to pre-load the image because it was just too slow > to > > load the individual fasl files at run time, either via asdf or through a > > generated load-list. > > > At ITA, we also deliver applications as dumped images with lots of systems. > But I admit that during incremental development, > we recompile files individually with SLIME > rather than load entire systems with ASDF. > I have a bad habit of just requring or load-system'ing whatever I'm working on. I sometimes have mutliple changes that i want to get loaded. > > > I'm using ASDF to load a small system at run-time, but finding that even > > with small system (say, one file), performance is sluggish, over 2 > seconds. > > > First, how big are that small system's dependencies? > Because they will all be scanned. > Second, how much time is spent in initialize-source-registry? > Because if you reset it when dumping an image > (which you probably should > unless the source is guaranteed to be at the same place at runtime), > it will scan all the declared source trees. > Well, a dependency might include one or more of my own systems, and then it goes into the big wonderful world of 3rd party depencies, like iterate, ppcre, base64, sqlite, etc. I haven't analysed initialize-source-registry, will do. I have not used asdf at run-time before -- haveing used a dumped image to run just a small hook file that starts pre-loaded functions, and then any other functions loaded after this do not declare any dependencies (via require or direct asdf invocation). This is a new thing for me. > > > It looks like the problem crops up when there are many dependencies. > Poking > > around the asdf code this appears to be because operate always traverses > the > > system dependencies, and pokes the asd files to find if they have > changed. > > > > This makes sense for asdf in development mode, since it needs to discover > > changed systems. Not so good for a deployed system. > > > Well, what do you propose ASDF should do? > Have some option to operate and/or traverse > to skip checking some dependencies if they are already loaded, > kind of a :force-not complement to :force ? > I suppose I'll accept a patch that does it. > In an abstract sense, it would be great if ASDF had a sense of target environment or strategy, I had a clever word for it before my kids came home and my brain went out the window. This would be development, deployment, testing, etc. and could wrap up things like the source, destintation directories as well as files storage types. E.g. it might be good to be able to package up stuff in zip files for delivery and have ASDF look inside them (with proper implementation of that file storage type.) The target environment could be used to specialize the behavior of asdf, for instance, how asdf deals with hunting down dependencies, dealing with no source files. It may be that the object system would already allow this by specializing operations and components, but it seems that the concept is orthogonal, no? This might even be a way at getting at the problem of searching the entire depency tree for upwind changes. > > > ASDF assumes that since the system def file has not changed, the system > > hasn't been changed either (I know there has been discussion of this > > assumption.) But lets assume that at least part of the reason for that is > > because it is very time consuming to check the entire set of files every > > time the developer wants to recompile and load just a single file. > > > I don't think that's true, but my head hurts just thinking about it. > > > > Why not assume in a deployment mode that NO system will have changed if > it > > is already loaded in memory, and unless the application has asked that > it be > > reloaded? This would be for performance as well as for safety -- we > really > > don't want dependency checking to result in any filesystem checks, etc. > > > Because that's wrong? I don't have as many dependencies as you do when > I reload a system with asdf, but usually, I do want the dependencies > to be reloaded. > Assume you have a couple dozen dependencies managed under quicklisp. These files are really hardly every updated, other than quicklisp batch update. There should be a way of ensuring that once loaded they are never consulted again during that lisp session (or in that saved image.) > > > I feel that this problem is really pushing my grokking of asdf, but I'm > > delving ahead anyway. > > > > I haven't tested it extensively, but I did just waste a perfect sunday > > morning fiddling with this: > > > > extended module with a slot "up-to-date-p", but surely there is a better > > name, this I just chose while in early play mode: > > > > added these methods: > > > > (defmethod operation-done-p ((o load-op) (c module)) > > (module-up-to-date-p c)) > > > > (defmethod mark-operation-done ((operation load-op) (c module)) > > (setf (module-up-to-date-p c) t)) > > > > tweaked the operate method so that it exits early if the module has been > > marked as up-to-date > > > > (if (operation-done-p op system) > > (return-from operate (values op nil))) > > > > and do-traverse to only collect "kids" if it is not done > > > > (unless (operation-done-p operation c) > > (while-collecting (internal-collect) > > > I think that's quite wrong. > How are you to maintain the up-to-date-ness of your modules? > Yes, this is just to make "an asdf which works the way I want it to in this situation". Remember, this is for a short-lived lisp session which just needs to load a system (module) once, and to never reload an already loaded system (because there are many preloaded systems in the image.) Also, for long lived deployed sessions I'm not at all interested in reloading systems through asdf. I do have functions that may need to be reloaded, but that is through a much simpler process of loading and compiling a function and storing it in a slot somewhere. In this case I was trying to use the concept of "done-ness", since it is already there in asdf. I consider a module loading to be done ... after it is loaded. > > > Note that this code goes in the special section of operate for handling > > modules. > > > > I think that is all that I did. > > > > I found that the time to handle the require for my current project > dropped > > from about 2 1/2 to about 1/2 second. > > > I'd rather you implement a :force-not. It sounds cleaner to me. > Ok, I'll explore that. > > > Thoughts? > > > I imagine a far future where an XCVB daemon watches changes to the > filesystem > to detect which registered builds require an update. Not quite there yet. > I'm very interested in the XCVB project, and this capability sounds great, but at the moment I'm working with a ton of asdf enlightened modules, including all of quicklisp, and am really interested in getting it to work optimally. Erik. > > —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• > http://fare.tunes.org > The older I grow, the more I distrust the familiar doctrine that age > brings wisdom. — H.L. Mencken > (Plus je vieillis, moins je crois en l'idée commune que l'age apporte > la sagesse.) >

1 0

[asdf-devel] Faster loading for deployed systems?
by Erik Pearson 22 Apr '12

22 Apr '12

Hi, I am trying to implement a method of using a "batteries included" lisp image (ccl, linux, 64bit) which can be used for running a variety of lisp tasks in a batch mode. I chose to pre-load the image because it was just too slow to load the individual fasl files at run time, either via asdf or through a generated load-list. I'm using ASDF to load a small system at run-time, but finding that even with small system (say, one file), performance is sluggish, over 2 seconds. It looks like the problem crops up when there are many dependencies. Poking around the asdf code this appears to be because operate always traverses the system dependencies, and pokes the asd files to find if they have changed. This makes sense for asdf in development mode, since it needs to discover changed systems. Not so good for a deployed system. ASDF assumes that since the system def file has not changed, the system hasn't been changed either (I know there has been discussion of this assumption.) But lets assume that at least part of the reason for that is because it is very time consuming to check the entire set of files every time the developer wants to recompile and load just a single file. Why not assume in a deployment mode that NO system will have changed if it is already loaded in memory, and unless the application has asked that it be reloaded? This would be for performance as well as for safety -- we really don't want dependency checking to result in any filesystem checks, etc. I feel that this problem is really pushing my grokking of asdf, but I'm delving ahead anyway. I haven't tested it extensively, but I did just waste a perfect sunday morning fiddling with this: extended module with a slot "up-to-date-p", but surely there is a better name, this I just chose while in early play mode: added these methods: (defmethod operation-done-p ((o load-op) (c module)) (module-up-to-date-p c)) (defmethod mark-operation-done ((operation load-op) (c module)) (setf (module-up-to-date-p c) t)) tweaked the operate method so that it exits early if the module has been marked as up-to-date (if (operation-done-p op system) (return-from operate (values op nil))) and do-traverse to only collect "kids" if it is not done (unless (operation-done-p operation c) (while-collecting (internal-collect) Note that this code goes in the special section of operate for handling modules. I think that is all that I did. I found that the time to handle the require for my current project dropped from about 2 1/2 to about 1/2 second. Thoughts?

3 2

[asdf-devel] Testing with RMCL....
by Robert Goldman 22 Apr '12

22 Apr '12

I am willing to run the ASDF tests using RMCL, since my Mac still has Snow Leopard, instead of Lion. *However*, as a non-RMCL user, I have no idea how to run it from the shell, so that we can run the test suite on RMCL. Can someone please offer me a clue? I can't be trying to run this from the GUI every time it needs a test. I had to give up testing LispWorks for the same reason. thanks, r

4 8

[asdf-devel] Please test 2.20.16
by Faré 21 Apr '12

21 Apr '12

OK, so I propose we freeze the current state of ASDF for a release 2.21, hopefully some time next week. Please test 2.20.16. Changes since 2.20: * New features: * Most importantly, encodings. ASDF now lets you specify an :encoding for a system, module or component, that is used when loading or compiling Lisp files. See the documentation. By default, the only useful value is :utf-8, and we recommend you use UTF-8 everywhere. We intend to make it the default in the future (current default is the legacy behavior of using whichever implicit default your underlying implementation is currently configured to use). An extension asdf-encodings is available that supports more encodings, including autodetection of encoding in the near future. * You can now specify :hostname in your asdf-output-translations, so you can easily share a home directory yet split its fasl cache between several subtly different machines. * Bug Fixes: * lp#982285. since 2.014.4, the default source-registry was skipping anything from /usr or XDG_DATA_DIRS, because of a missing test in getenv-absolute-pathnames. Fixed in .15. * In 2.019.9 aka 2.20, ECL support was broken by using a function in the wrong package (defined in cl-user). Fixed in .1. * logical-pathname support was somewhat broken since 2.017.6, due to the way most implementations fail to read physical namestrings when *default-pathname-defaults* is a logical-pathname. Fixed, together with other logical-pathname issues, and a test case was added to the test suite to ensure no further regression. Works great modulo quirks around implementation bugs on CLISP and Allegro. * Minor tweaks: * Use :unspecific in pathname components on more implementations. * export and/or document more utilities. * add a few missing compatfmt for Genera. I do not intend to make any further changes to ASDF before the 2.21 release, except for fixes to any bug found, of course. Experiments with encodings and autodetection are redirected to the asdf-encodings system for the time being. Discussion of a possible merge of asdf-encodings into ASDF will happen only after asdf-encodings itself is reasonably stable. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The mice which helplessly find themselves between the cats teeth acquire no merit from their enforced sacrifice. — Mahatma Gandhi

4 14

[asdf-devel] ASDF and character encodings
by Daniel Herring 21 Apr '12

21 Apr '12

This topic came up at the Boston Lisp meeting last night. Here was my takeaway from the discussion. ASDF does not need to support arbitrary encodings in system definition files. UTF-8 is the simplest and most portable solution for non-ASCII characters in system definitions. Thus ASDF can simply declare that all system definitions are read using UTF-8. On systems where unicode support is not available, system definitions will be read as "extended ASCII", and the non-ASCII characters will simply print incorrectly. Filenames should only be specified using ASCII characters to avoid portability issues. Authors do need a way to support arbitrary encodings in other source files, but this should be restricted to cases where there is an actual need to use a non-UTF-8 character set. Example use cases are locale-specific libraries, a toolchain that converts between character sets, or possibly a character-set-detection library. Again, other character sets can be a major portability issue. There are only a handful of libraries that use something other than ASCII or UTF-8. These should be easy to fix, given approval from their authors. Hopefully that's an accurate record of the general consensus and this writeup will be helpful to others in some way. - Daniel

2 2

[asdf-devel] Tested all on 2.20.12 Mac OS X 10.6
by Robert Goldman 18 Apr '12

18 Apr '12

Seems to work fine. Still not testing on CMUCL, waiting for patch to that. cheers, r

3 4

[asdf-devel] Ran the tests on Mac OS X
by Robert Goldman 16 Apr '12

16 Apr '12

...all have passed. rpg% echo $ASDF_TEST_LISPS ccl clisp sbcl ecl abcl allegro allegromodern Cheers, r

1 0

[asdf-devel] source file encoding
by Anton Vodonosov 16 Apr '12

16 Apr '12

Hello. How to specify source files encoding in ASD2? I.e. what value to pass to the :external-format argument of the compile-file? In the old ASDF I did it by (let ((asdf:*compile-file-external-format* #+:clisp charset:cp1251 #+:sbcl :cp1251)) (asdf:operate 'asdf:load-op :mysystem)) But today the variable asdf:*compile-file-external-format* does not exist. Also, it of course would be better to specify the encoding in the .asd file (the system author knows the files encoding, and should care about this, but not the system user). Could you advise me how to do it? Best regards, - Anton

13 61

[asdf-devel] logical pathname fixes
by Faré 13 Apr '12

13 Apr '12

I added a more functional test for logical pathnames, and made a round of fixes for logical pathnames. Some things are still broken on CLISP and Allegro, but I blame the implementations (see test/test-logical-pathname.script for details). p-cos, janderson, pjb: I believe you are the three known users of logical pathnames. Can you test the latest ASDF in your respective setups? (currently 2.20.10). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Think you can, or think you can't — either way, you'll be right. — Henry Ford

2 6

[asdf-devel] What are current plans for RMCL?
by Pascal Costanza 12 Apr '12

12 Apr '12

Hi, I'd like to get an idea what the current plans for RMCL are, if any. Since the switch to OS X 10.7, there is no default support for old PowerPC applications on OS X anymore, so RMCL doesn't work anymore, at least not without a major effort. Are there any plans to find a remedy for this situation? Or will RMCL effectively become deprecated? I'm asking for the following reasons: When ASDF was changed from 1.x to 2.x, this caused some problems for RMCL, which I eventually resolved by using Common Lisp's logical pathnames for the systems I maintain (primarily Closer to MOP and ContextL). However, the current maintainers of ASDF have an unjustified very low regard for logical pathnames, which causes a lot of pain - basically, whenever a new version of a Common Lisp implementation comes bundled with a new ASDF version, I have to deal with bugs in ASDF that in one way or the other break my setup with logical pathnames. Since my time is limited and is better served on things other than producing bug reports for ASDF (which is a tool that should be much more stable than it currently is), I decided now that it is better to drop logical pathnames and go for the Unix-like names that the ASDF maintainers seem to strongly prefer. RMCL would be the only Common Lisp implementation that would cause a problem in this regard. So, what's the verdict? Thanks a lot for any hints! Best, Pascal -- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

3 7

[asdf-devel] Latest commit
by Robert Goldman 12 Apr '12

12 Apr '12

I just ran the noupgrade tests (with CMUCL disabled) on Mac OS X, and all passed. I assume that I don't need to bother with Linux testing.... r

1 0

[asdf-devel] testing new ASDF
by Robert Goldman 11 Apr '12

11 Apr '12

All tests pass on Mac OS X until I get to CMUCL 20c, which still fails for reasons that seem unrelated to ACL. These are test-all-noupgrade. Will test with upgrade later. Note that I am still running 10.6, not the latest 10.7. It would probably be a Good Thing to find someone who will run the tests on 10.7. For the moment, I'm purging CMUCL from my test suite. Cheers, r

1 0

[asdf-devel] LOCALAPPDATA vs APPDATA
by Faré 01 Apr '12

01 Apr '12

On Sun, Apr 1, 2012 at 14:35, Anton Vodonosov <avodonosov(a)yandex.ru> wrote: > 01.04.2012, 22:19, "Faré" <fahree(a)gmail.com>: >>> 15.01.2012, 22:46, "Faré" <fahree(a)gmail.com>: >>>> I'm a bit surprised that LOCALAPPDATA includes Roaming\ instead of >>>> Local\ but then again I don't understand Windows paths very well. >> >>> If you are intrested, I just noticed: CLISP uses Roaming, while SBCL and CCL use Local. >>> (noticed with the ASDF version from the recent Quicklisp) >> >> Wow. Is that on the *same* machine? >> Because on all implementations but LispWorks, >> it's taken from the LOCALAPPDATA environment variable, >> and that shouldn't vary from one program to the other. > > Yes, same machine. > > I suppose you mean APPDATA variable, because it is mentioned in asdf.lisp, > but LOCALAPPDATA isn't. Anyway, the enviroment variable value is equal > in all the 3 lisps > ASDF has favoring LOCALAPPDATA over APPDATA since 2.015.6. But I see quicklisp is still using 2.014.6. Now, considering the ECL bug in 2.20 (aka 2.019.9), I recommend that ECL and/or Quicklisp should upgrade to 2.019.8 (if they upgrade), until 2.21 is ready (which it won't be until we fix those 20 systems that depend on something else than UTF-8). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org (map()(lambda(n l)(princ(subseq (format nil "(~{~A~^ ~})~%"(let(r)(do-external-symbols(x :cl)(push x r))(sort r #'string< :key #'string))) n (+ n l)))(finish-output))'(875 949 8 6010 4863 371 11)'(4 3 5 5 3 2 2))

1 0

[asdf-devel] the same (asdf::implementation-identifier) for byte-compiler and lisp-to-C-compiler of ECL
by Anton Vodonosov 01 Apr '12

01 Apr '12

ECL offers two compiles: byte-compiler and lisp-to-C compiler. They produce incompatible .fasls (asdf::implementation-identifier) does not accounts the difference and returns the same value - ecl-11.1.1-win-x86. As implementation-identifier is used to compute output files location, the same file location is used to store .fasls of both ECL compilers (in my case C:\Users\anton\AppData\Roaming\common-lisp\cache\ecl-11.1.1-win-x86\). In result, if some files are compiled with byte-compiler, and latter user tries the lisp-to-C-compiler, he has errors during .fasl load.

2 5