asdf-devel

asdf-devel@common-lisp.net

5 participants
1375 discussions

[asdf-devel] Disabling "built-in" output translation
by Mark Evenson 21 Jun '11

21 Jun '11

ASDF contains some specialized code for ABCL that enable the translation of ASDF systems whose source is packaged into jar files to locate the corresponding object files in ASDF's user cache. In trying to figure out the correct mechanism to allow for "binary" ASDF distribution under ABCL that would include both the source and the fasls in the same jar file, I need to disable the jar file translation mechanism. I expected that invoking DISABLE-OUTPUT-TRANSLATIONS would clear this built-in translation mechanism, but it doesn't seem to do this. Only invoking CLEAR-OUTPUT-TRANSLATIONS seems to actually change the output translation map. To wit: ASDF> (disable-output-translations) ((T T) (#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (output-translations) ((#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T T) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (disable-output-translations) ((T T) (#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (output-translations) ((#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T T) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (clear-output-translations) ; No value ASDF> (output-translations) NIL ASDF> Should DISABLE-OUTPUT-TRANSLATIONS actually push that (T T) list into the beginning of that (OUTPUT-TRANSLATION) list or does the API expect that the user use it something like ASDF> (setf (output-translations) (disable-output-translations)) Should ASDF under ABCL strip out the jar translation mechanism as well, or should the ASDF API expect that the user would actual issue a (CLEAR-OUTPUT-TRANSLATIONS)? I confess that I am a bit confused when one would ever use DISABLE-OUTPUT-TRANSLATIONS as opposed to CLEAR-OUTPUT-TRANSLATIONS. Once I understand the ASDF API here a bit better, I will work on the necessary ABCL specific ASDF patches if necessary. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

2 6

Re: [asdf-devel] [Sbcl-devel] Logical pathnames vs ASDF & SBCL
by Zach Beane 17 Jun '11

17 Jun '11

"Scott L. Burson" <Scott(a)sympoiesis.com> writes: > On Mon, Jun 13, 2011 at 6:17 AM, Faré <fahree(a)gmail.com> wrote: >> I don't see why you believe that ASDF 1.x isn't a viable option. > > I'm not among those having trouble with ASDF v2, but I would just > point out that going back to v1 has been made rather unattractive by > the tremendous usefulness of Quicklisp, which AFAIK depends on v2. No, it doesn't. When I started sharing Quicklisp, some implementations didn't ship ASDF at all, so it made sense to bundle a copy and load it if necessary. ASDF2 seemed like the reasonable thing to bundle. The feature of ASDF2 that makes Quicklisp work better is implementation FASL segregation enabled by default. Zach

1 0

[asdf-devel] Logical pathnames vs ASDF & SBCL
by Faré 17 Jun '11

17 Jun '11

TL;DR: ASDF currently doesn't play so well with logical pathnames. I'm seeking advice on how to make things better — or on whether it's worth the trouble. Dear all, Pascal's recent plight with logical pathnames on SBCL once again rises the issue of how much ASDF (or SBCL) should support logical pathname users, and how they should behave to make logical pathname users happy. With case-sensitivity issues, things only get worse! (Note: personally inviting James Anderson to the discussion, since he's the only strong proponent and user of logical pathnames I know.) REMINDER ABOUT LOGICAL PATHNAMES (LPNs) So, the CLHS says LPNs are case-converted to upper-case, portably restricted to such upper-case characters, and get translated to physical pathnames in an implementation-dependent way. On Unix, with its case-sensitive filesystems and lower-case conventions, the useful thing is then downcasing names. At least SBCL and CLISP follow both the standard and the useful pathname mapping convention. SBCL goes on to length to throw an error when non-portable code tries to break the standard restrictions on logical-pathnames; CLISP will let you manually make-pathname a LPN with lower-case or mixed-case characters, but will still downcase them at the end (the latest CLISP 2.49 also seems to not want to create a logical pathname with parse-namestring). CCL tries to make users happy by leaving the logical pathname host case-insensitive (per the standard) but case-preserving, and being case-sensitive in the rest of the pathname (if you want upper-case, you know where to find it). To collect and analyze the behavior of each of 14 implementations is left as an exercise to the logical-pathname loving reader [M20]. To devise, document and get implementations to adopt a common behavior that would be universally accepted by all is left as another exercise [50]. ASDF NAMING OF SYSTEMS Similarly, when naming an ASDF system, to allow to designate systems with symbols (CL legacy), which are case-converted, yet map them to a unixy lower-case name, and still allow systems with all kind of cases, symbols are downcased whereas strings preserved. Then the filesystem is queried with the resulting normalized string name (see function coerce-name). Now with its good old central-registry, ASDF used to query each directory every time a system was requested. This works great with logical pathnames, as it defers all case handling to the last minute, and allows .asd files to be loaded with a LPN as the *load-pathname*. However, this was somewhat slow, and it didn't scale to recursively searching directories, and so came the habit of creating "link farms", single directories with plenty of symlinks to adequate .asd files. Of course, symlinks are both not so portable (not very well supported in Windows, if at all, though later versions of ASDF recognize Windows shortcuts as a substitute), and a maintenance nightmare (unlike Macintosh aliases, they are not automatically updated when a system is installed, moved or deleted). Therefore link farms were not a good interface to end-users. For ASDF 2, I implemented a new way to manage system files: the source-registry. It allows you to specify trees to search recursively as well as single directories. Whether the registry is searched lazily (when a system is requested) or eagerly (one the first time around) or yet something else, was left unspecified. In my first implementation of it, I built upon the previous central-registry, and at initialization time I was looking for which subdirectories in the specified trees did contain .asd files, then saved these in a list that I consulted in the very same way as the central-registry. (Note: the central registry is still there, and consulted first.) However, DIRECTORY returns fully resolved TRUENAMEs, and any portable way of searching inside subdirectories is doomed to squash away logical pathnames. Currently, any use of :tree in source-registry will resolve into physical pathnames (PPNs) on most implementations, and always did. However, entries with :directory used to preserve LPNs. Not so with 2.014.7, where, prompted by a Quicklisp feature request, I implemented eager caching of ASDF systems. So I query DIRECTORY and save the results in a hash-table indexed by pathname-name. The table is case-sensitive, to reflect the case-sensitive names of ASDF. One consequence is that :directory entries do not preserve LPNs anymore. Another more subtle consequence is that things may go wrong if your filesystem is case-insensitive but case-preserving. In the old way of doing things (still available with the central-registry), you'd typically be looking for the file with lower-case name, and you would find it even if named in different cases on disk. With the old way, you'll record the name in its case mix, and will never match it against the (lowercase) name. So some systems that depend on this behavior will disappear. POTENTIAL SOLUTIONS An easy way out is to declare that ASDF from now on shall be case-insensitive, using EQUALP to compare system names and in hash-tables that cache system files. Another way out is to declare that ASDF is still case sensitive, and that the only case allowed is lower-case for physical filenames. Any system that doesn't work with this new interpretation probably didn't work portably with the old one, anyway. A harder way out would be to assume case-sensitivity in the filesystem (since some filesystems are case-sensitive, and any asd file that depends on case-insensitivity is non-portable and deserves to lose), and make extra effort when using DIRECTORY against logical pathnames to always reconstitute LPNs from the PPNs given by directory, and drop the reconstituted results if they don't resolve to the same thing. But is it worth the pain? Finally, I could do nothing, and assume that people who go through the pain of setting up logical pathnames before they run ASDF can just as well setup the *central-registry* and have LPNs be recognized in the good old way. If you use the source-registry, your LPNs will be squashed, and nothing will break in the code since code works fine with PPNs, just like most everyone use it, but your debug information will be stored according to the PPN, not the LPN. Meh. Zach (if you're still reading), do any systems in the wild use anything but lower-case names for system files? [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Two members of the Political Police salute a giant portrait of our Great Leader - Comrade, do you think the same as I do? inquires the first one, with a sincere gaze pleeing into his colleagues' eyes. - Comrade, the second one replies and returns the same gaze: yes, I do. - In that case, comrade, you're under arrest! concludes the first one.

7 10

[asdf-devel] pathname case issue ...
by Chun Tian (binghe) 17 Jun '11

17 Jun '11

Hi, ASDF developer Recently I found a strange behaivour in latest ASDF: it internally converts all my pathname namestrings into lower-case, although I'm using a case-insentive HFS on Mac. I have following items in asdf:*central-registry*, these packages are either maintained/modified by me or not yet with Quicklisp: SNMP 14 > asdf:*central-registry* (#P"/Applications/FileMaker Pro 11/fm-plugin-tools-0.2.9/" #P"/Users/binghe/Lisp/packages/trivial-gray-streams/" #P"/Users/binghe/Lisp/packages/lisa/" #P"/Users/binghe/Lisp/packages/elephant/" #P"/Users/binghe/Lisp/packages/cffi/" #P"/Users/binghe/Lisp/packages/cl-base64/" #P"/Users/binghe/Lisp/packages/iolib/src/" #P"/Users/binghe/Lisp/cl-net-snmp/snmp/branches/6/" #P"/Users/binghe/Lisp/usocket/branches/0.5.x/" #P"/Users/binghe/quicklisp/quicklisp/") Notice that all of those pathnames except the first one is started from directory "/Users/binghe", in which the letter "U" is upper-cased. But, when I manually compiled a changed definition in one of above projects (snmp), I got a warning from my Lisp platform: """ Warnings: (DEFUN SNMP-CONNECT) being redefined in /users/binghe/Lisp/cl-net-snmp/snmp/branches/6/session.lisp (previously in /Users/binghe/Lisp/cl-net-snmp/snmp/branches/6/session.lisp). """ See, now ASDF think the Lisp file was in another different directory which started from "/users/binghe/, in which the first letter "u" is lower-cased. I hate this new behaviour, either because I want to keep all my namestring cases in a case-insentive file system, or I doubt that if I'm in a case-sensitive HFS filesystem, ASDF may failed to distinguish /Users and /users or something like this. Please confirm it's a new feature or bug and please consider reverting to previous status. Regards, Chun Tian (binghe)

5 16

[asdf-devel] Force reloading a .asd file without "touch" it
by Chun Tian (binghe) 15 Jun '11

15 Jun '11

Hi, ASDF users Any one know how to reload a .asd file without "touch" it to let it have a newer timestamp? I have a snmp.asd file which contain the content of another file: (defparameter *mib.lisp-expr* (with-open-file (s (merge-pathnames (make-pathname :name "mib" :type "lisp-expr") *load-truename*) :direction :input) (read s))) (defsystem snmp ... :components ( ... (:module "compiled-mibs" :depends-on ("runtime") :components #.*mib.lisp-expr*))) See, the DEFSYSTEM form is actually determined by another file. And I have a function which can update this another file, and after its content change, I want to re-load this snmp.asd to make the new DEFSYSTEM form take effect. Just call (asdf:load-system :snmp) obviously won't help. How can I achieve this goal? Thanks, Chun Tian (binghe)

2 2

[asdf-devel] ASDF 2.016.2
by Faré 13 Jun '11

13 Jun '11

Regarding RMCL and logical pathnames, please test 2.016.2. I introduce some non-trivial pathname changes. On RMCL, ASDF sees strings as "/posix/namestrings/" in its configuration. if you want OS9-style pathnames, they are #p"that:a:way:". You can thus both trivially share your configuration with other implementations, and just as trivially use MCL-specific paths. For logical pathnames, I implemented my "harder way" plan from previous email: > make extra effort when using DIRECTORY against logical pathnames > to always reconstitute LPNs from the PPNs given by directory, and > drop the reconstituted results if they don't resolve to the same thing. In other words, if I compute (directory #p"FOO:CL;BAR;"), and find a file #p"/home/baz/cl/bar/quux.asd", the file will be kept only if (make-pathname :name "quux" :type "asd" :defaults #p"FOO:CL;BAR;") has the previous as its truename. And similarly when walking directories. I believe this restriction is not a problem in practice, and will keep things somewhat sane. And no, it's probably still not a good idea to try mix logical pathnames and mixed-case physical pathnames, as the result is highly non-portable. Also removed *system-cache* and getuid, which were bad ideas that never worked (and if anyone is using either, he'll thank me for making him stop). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I believe in God, only I spell it Nature. — Frank Lloyd Wright

4 7

[asdf-devel] clear-output-translations and saved cores
by Raymond Toy 13 Jun '11

13 Jun '11

Xach brought this to my attention wrt to maxima, but it's not really about maxima and asdf2. My question is if clear-output-translations should always be called before saving a core. Is there a case where you would not want to run c-o-t before saving a core? If c-o-t should always be run, wouldn't it make sense for asdf2 to arrange for that? I'm not familar with how other lisps do this, but cmucl does have *before-save-initializations* which is a list of functions that are run before a core is saved. Perhaps asdf2 should push #'c-o-t onto *before-save-initializations*? Ray

4 6

[asdf-devel] SBCL 1.0.49 / ASDF 2.015.3 doesn't load my asdf systems anymore
by Pascal Costanza 12 Jun '11

12 Jun '11

Hi, See subject line. Here is a trace from the listener: This is SBCL 1.0.49, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. * (asdf:load-system :closer-mop) debugger invoked on a ASDF:MISSING-COMPONENT: Component "closer-mop" not found Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [ABORT] Exit debugger, returning to top level. ((LAMBDA ())) 0] :backtrace 0: ((LAMBDA ())) 1: ((LAMBDA ())) 2: (ASDF::CALL-WITH-SYSTEM-DEFINITIONS #<CLOSURE (LAMBDA #) {1003E98CA9}>) 3: ((SB-PCL::FAST-METHOD ASDF:OPERATE (T T)) #<unused argument> #<unused argument> ASDF:LOAD-OP :CLOSER-MOP) 4: ((SB-PCL::EMF ASDF:OPERATE) #<unused argument> #<unused argument> ASDF:LOAD-OP :CLOSER-MOP) 5: (ASDF:LOAD-SYSTEM :CLOSER-MOP) 6: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ASDF:LOAD-SYSTEM :CLOSER-MOP) #<NULL-LEXENV>) 7: (EVAL (ASDF:LOAD-SYSTEM :CLOSER-MOP)) 8: (INTERACTIVE-EVAL (ASDF:LOAD-SYSTEM :CLOSER-MOP) :EVAL NIL) 9: (SB-IMPL::REPL-FUN NIL) 10: ((LAMBDA ())) 11: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #<CLOSURE (LAMBDA #) {1003E982F9}>) 12: (SB-IMPL::TOPLEVEL-REPL NIL) 13: (SB-IMPL::TOPLEVEL-INIT) 14: ((FLET #:WITHOUT-INTERRUPTS-BODY-[RESTART-LISP]30)) 15: ((LABELS SB-IMPL::RESTART-LISP)) 0] This happens with other systems as well. (I haven't been able to load any system anymore, but I haven't tried all that I have installed.) What information do you need from me to be able to fix this? Best, Pascal -- Pascal Costanza The views expressed in this email are my own, and not those of my employer.

3 14

[asdf-devel] Issue in GETUID (MCL, again) and others
by Chun Tian (binghe) 11 Jun '11

11 Jun '11

Hi, Faré Today I compile latest asdf.lisp in MCL and see two warnings: ? (compile-file #P"Macintosh HD:Users:binghe:Lisp:packages:asdf:asdf.lisp") ;Compiler warnings for "Macintosh HD:users:binghe:Lisp:packages:asdf:asdf.lisp" : ; :LAMBDA compiler warning with args (&OPTIONAL), in WITH-SYSTEM-DEFINITIONS. ;Compiler warnings for "Macintosh HD:users:binghe:Lisp:packages:asdf:asdf.lisp" : ; Undefined function CCL::GETUID, in ASDF::GET-UID. #P"Macintosh HD:users:binghe:Lisp:packages:asdf:asdf.cfsl" T T The warning "Undefined function CCL::GETUID, in ASDF::GET-UID" was caused by #+ccl in following piece of code: #+asdf-unix (progn #+ecl #.(cl:and (cl:< ext:+ecl-version-number+ 100601) '(ffi:clines "#include <sys/types.h>" "#include <unistd.h>")) (defun* get-uid () #+allegro (excl.osi:getuid) #+ccl (ccl::getuid) ... Both Clozure CL and MCL contains the feature :CCL, so the form (ccl::getuid) was wrongly exposed to MCL. I think #+clozure (or legacy #+openmcl) should be used instead, just like other places in asdf.lisp. (if run-shell-command works on MCL, the function GET-UID should also work in MCL) A trivial patch is in attach (there's another #-ccl, don't forget it) P. S. For the first warning, why macro with-system-definitions have a standalone &optional in its parameter list? (defmacro with-system-definitions ((&optional) &body body) `(call-with-system-definitions #'(lambda () ,@body))) Regards, Chun Tian (binghe)

2 1

[asdf-devel] ASDF 2.016 breaks ABCL translations for jar files
by Mark Evenson 10 Jun '11

10 Jun '11

Stellian's [normalization to ANSI semantics of COMPILE-FILE-PATHNAME*][1] has the unfortunate effect of breaking ABCL's translation of systems packaged in jar files. [1]: http://common-lisp.net/gitweb?p=projects/asdf/asdf.git;a=commit;h=ed4cd3293… I have applied the subsequent patch to the asdf-2.016 packaged with ABCL to restore our needed functionality but am unhappy at the necessity of introducing a runtime implementation conditional, as sure a sign as anything that I'm "doing it wrong," and can't really request that ASDF patch itself in this manner. Maybe if I explain things as I have analyzed them, someone else can suggest a better way forward. Recall that ABCL uses the presence of a cons in the DEVICE component to indicate that a Pathname represents an entry in a jar archive. A pathname located within a jar archive is not a writable location, so we introduced ABCL specific ASDF output translation rules which translate such a source location to an appropriately differentiated output location in the ASDF cache hierarchy. This results in ASDF making the following kind of call in preparation of the output file location it will pass to COMPILE-FILE: (compile-file-pathname* <path with a DEVICE> :output-file <path without a DEVICE>) Stellian's patch correctly fixes behavior of COMPILE-FILE-PATHNAME* to use pathname components in input-file if none are specified in output-file. For ABCL systems in a jar, this results in returning an output location within the jar file, which being a non-valid output location because it is not writable causes ASDF to error out in compiling the system. If there were some mechanism to indicate that the invocation of APPLY-OUTPUT-TRANSLATIONS had really "finished" so that no further processing was necessary, we could possibly plausibly short-circuit this call to COMPILE-FILE-PATHAME*. I suppose my recursive invocation of APPLY-OUTPUT-TRANSLATIONS in the ABCL-specific TRANSLATE-JAR-PATHNAME adds the missing TYPE to the pathname which other code paths don't have at this point, which is why the call to COMPILE-FILE-PATHNAME* is necessary. ANSI allows the addition of implementation specific arguments to COMPILE-FILE-PATHNAME, so we could maybe add a :strip-device for ABCL but this seems even less elegant. Any suggestions? @@ -3523,7 +3525,16 @@ --- a/asdf.lisp Thu Jun 09 06:03:20 2011 +0000 +++ b/asdf.lisp Thu Jun 09 15:00:45 2011 +0200 (defun* compile-file-pathname* (input-file &rest keys &key output-file &allow-other-keys) (if (absolute-pathname-p output-file) - (apply 'compile-file-pathname (lispize-pathname input-file) keys) + ;;; If the default ABCL rules for translating from a jar path to + ;;; a non-jar path have been affected, no further computation of + ;;; the output location is necessary. + (if (and (find :abcl *features*) + (pathname-device input-file) ; input-file is in a jar + (not (pathname-device output-file)) ; output-file is not in a jar + (equal (pathname-type input-file) "lisp") + (equal (pathname-type output-file) "abcl")) + output-file + (apply 'compile-file-pathname (lispize-pathname input-file) keys)) (apply-output-translations (apply 'compile-file-pathname (truenamize (lispize-pathname input-file)) -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

2 11

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

asdf-devel