I see the following when I try to load an asdf system now:
Error loading #P"/Users/rpg/clinit.cl": While searching for system `asdf-context`: `(MERGE-PATHNAMES systems/ /Users/rpg/lisp/asdf-install-systems/)` evaluated to `/Users/rpg/lisp/asdf-install-systems/systems/` which is not a directory.
The problem with this error is that, in fact, that IS a directory:
CL-USER(2): (probe-file "/Users/rpg/lisp/asdf-install-systems/systems/") #P"/Users/rpg/lisp/asdf-install-systems/systems/"
I'm looking in sysdef-central-registry-search and the logic seems to be completely borked. It looks for a system definition and, if it doesn't find one, it assumes that the directory name is bad. I believe the logic should be:
1. Look at the directory and ensure that it exists.
2. If it does not exist, push it onto the bad list
3. Else merge in the system name and look for the system definition.
The current logic merges in the system name first, so misdiagnoses.
[I will now proceed to wrestle with git, which I am coming to loathe with a consuming hatred, in order to attempt to verify that my diagnosis is correct, and then send a patch. More soon.]
Hi Robert
Thanks for looking into this.
-- Earth First (we'll take care of the other planets later)
On Jul 13, 2009, at 12:10 PM, Robert Goldman rpgoldman@sift.info wrote:
I see the following when I try to load an asdf system now:
Error loading #P"/Users/rpg/clinit.cl": While searching for system `asdf-context`: `(MERGE-PATHNAMES systems/ /Users/rpg/lisp/asdf-install-systems/)` evaluated to `/Users/rpg/lisp/asdf-install-systems/systems/` which is not a directory.
The problem with this error is that, in fact, that IS a directory:
CL-USER(2): (probe-file "/Users/rpg/lisp/asdf-install-systems/ systems/") #P"/Users/rpg/lisp/asdf-install-systems/systems/"
I'm looking in sysdef-central-registry-search and the logic seems to be completely borked. It looks for a system definition and, if it doesn't find one, it assumes that the directory name is bad. I believe the logic should be:
Look at the directory and ensure that it exists.
If it does not exist, push it onto the bad list
Else merge in the system name and look for the system definition.
The current logic merges in the system name first, so misdiagnoses.
[I will now proceed to wrestle with git, which I am coming to loathe with a consuming hatred, in order to attempt to verify that my diagnosis is correct, and then send a patch. More soon.]
asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
Robert Goldman wrote:
I see the following when I try to load an asdf system now:
Error loading #P"/Users/rpg/clinit.cl": While searching for system `asdf-context`: `(MERGE-PATHNAMES systems/ /Users/rpg/lisp/asdf-install-systems/)` evaluated to `/Users/rpg/lisp/asdf-install-systems/systems/` which is not a directory.
The problem with this error is that, in fact, that IS a directory:
CL-USER(2): (probe-file "/Users/rpg/lisp/asdf-install-systems/systems/") #P"/Users/rpg/lisp/asdf-install-systems/systems/"
I'm looking in sysdef-central-registry-search and the logic seems to be completely borked. It looks for a system definition and, if it doesn't find one, it assumes that the directory name is bad. I believe the logic should be:
Look at the directory and ensure that it exists.
If it does not exist, push it onto the bad list
Else merge in the system name and look for the system definition.
The current logic merges in the system name first, so misdiagnoses.
[I will now proceed to wrestle with git, which I am coming to loathe with a consuming hatred, in order to attempt to verify that my diagnosis is correct, and then send a patch. More soon.]
OK, I /have/ misdiagnosed this. The logic seems actually busted in directory-pathname-p. The problem is that, at least on allegro, you can get a valid directory pathname whose name component is neither NIL, nor :unspecific, but "" (the empty string).
This seems like a place where wrestling with CL pathnames is very unpleasant.... What I'd really like to do here is resolve the pathname and just ask the bloody OS if this is a directory or not, instead of do these kind of seemingly random flailings with the pathname object:
(defun directory-pathname-p (pathname) (and (member (pathname-name pathname) (list nil :unspecific)) (member (pathname-type pathname) (list nil :unspecific))))
This will now have to be
(member (pathname-name pathname) (list nil "" :unspecific) :test 'equal)
which seems like it's metastasizing complexity...
Robert Goldman wrote:
Robert Goldman wrote:
I see the following when I try to load an asdf system now:
Error loading #P"/Users/rpg/clinit.cl": While searching for system `asdf-context`: `(MERGE-PATHNAMES systems/ /Users/rpg/lisp/asdf-install-systems/)` evaluated to `/Users/rpg/lisp/asdf-install-systems/systems/` which is not a directory.
The problem with this error is that, in fact, that IS a directory:
CL-USER(2): (probe-file "/Users/rpg/lisp/asdf-install-systems/systems/") #P"/Users/rpg/lisp/asdf-install-systems/systems/"
I'm looking in sysdef-central-registry-search and the logic seems to be completely borked. It looks for a system definition and, if it doesn't find one, it assumes that the directory name is bad. I believe the logic should be:
Look at the directory and ensure that it exists.
If it does not exist, push it onto the bad list
Else merge in the system name and look for the system definition.
The current logic merges in the system name first, so misdiagnoses.
[I will now proceed to wrestle with git, which I am coming to loathe with a consuming hatred, in order to attempt to verify that my diagnosis is correct, and then send a patch. More soon.]
OK, I /have/ misdiagnosed this. The logic seems actually busted in directory-pathname-p. The problem is that, at least on allegro, you can get a valid directory pathname whose name component is neither NIL, nor :unspecific, but "" (the empty string).
This seems like a place where wrestling with CL pathnames is very unpleasant.... What I'd really like to do here is resolve the pathname and just ask the bloody OS if this is a directory or not, instead of do these kind of seemingly random flailings with the pathname object:
(defun directory-pathname-p (pathname) (and (member (pathname-name pathname) (list nil :unspecific)) (member (pathname-type pathname) (list nil :unspecific))))
This will now have to be
(member (pathname-name pathname) (list nil "" :unspecific) :test 'equal)
which seems like it's metastasizing complexity...
Yet worse. On the #lisp IRC antifuchs and Xach point out more possible problems:
antifuchs: actually, a :name component of "." with :type nil/:unspecific would make sense in posix
Xach: antifuchs: or :name nil and :type "" :-)
This suggests that fixing directory-pathname-p may be sufficiently horrible that we should back out this attempted helpfulness.
The only alternative seems to me to be to figure out a way to ask the OS whether a file name denotes a directory, which seems like such an expensive thing that we may not want ASDF to do it inside an inner loop like sysdef-central-registry-search.
Every time I look deeply into CL pathnames I regret them more....
OK, I /have/ misdiagnosed this. The logic seems actually busted in directory-pathname-p. The problem is that, at least on allegro, you can get a valid directory pathname whose name component is neither NIL, nor :unspecific, but "" (the empty string).
Ugh,
(member (pathname-name pathname) (list nil "" :unspecific) :test 'equal)
A source of the problem are pathnames that arise from things like this:
> (make-pathname :name "" :directory '(:absolute "tmp")) #p"/tmp/" > (pathname-name *) ""
Added a call to namestring seems to be another way to canonical things so I think this will also work and feels (to me) a bit more portable:
(defun directory-pathname-p (pathname) (let ((pathname (namestring pathname))) (and (member (pathname-name pathname) (list nil :unspecific)) (member (pathname-type pathname) (list nil :unspecific)))))
wadayathink? Should I push? -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
Gary King writes:
OK, I /have/ misdiagnosed this. The logic seems actually busted in directory-pathname-p. The problem is that, at least on allegro, you can get a valid directory pathname whose name component is neither NIL, nor :unspecific, but "" (the empty string).
Ugh,
(member (pathname-name pathname) (list nil "" :unspecific) :test 'equal)
A source of the problem are pathnames that arise from things like this:
> (make-pathname :name "" :directory '(:absolute "tmp")) #p"/tmp/" > (pathname-name *) ""
Added a call to namestring seems to be another way to canonical things so I think this will also work and feels (to me) a bit more portable:
(defun directory-pathname-p (pathname) (let ((pathname (namestring pathname))) (and (member (pathname-name pathname) (list nil :unspecific)) (member (pathname-type pathname) (list nil :unspecific)))))
I see two problems:
(1) If the the host component of *DEFAULT-PATHNAME-DEFAULTS* is different than the host component of PATHNAME, the parse might fail or come out wrong.
(2) Pathnames with "" for the name component don't have namestrings under SBCL. (IIUC, the intent there is to try to have namestring parsing and unparsing be non-lossy.)
I think it might do just as well to say that "" for the name counts the same as NIL or :UNSPECIFIC for DIRECTORY-PATHNAME-P, unless somebody knows of implementations that don't work that way.
-- RmK
Gary King wrote:
OK, I /have/ misdiagnosed this. The logic seems actually busted in directory-pathname-p. The problem is that, at least on allegro, you can get a valid directory pathname whose name component is neither NIL, nor :unspecific, but "" (the empty string).
Ugh,
(member (pathname-name pathname) (list nil "" :unspecific) :test 'equal)
A source of the problem are pathnames that arise from things like this:
> (make-pathname :name "" :directory '(:absolute "tmp"))
#p"/tmp/" > (pathname-name *) ""
Added a call to namestring seems to be another way to canonical things so I think this will also work and feels (to me) a bit more portable:
(defun directory-pathname-p (pathname) (let ((pathname (namestring pathname))) (and (member (pathname-name pathname) (list nil :unspecific)) (member (pathname-type pathname) (list nil :unspecific)))))
wadayathink? Should I push?
Will this work correctly with logical pathnames?
The discussion on #lisp suggests a new wrinkle: are we trying to correct what are pathnames that incorrectly trying to name directories (this is what directory-pathname-p attempts unsuccessfully to do), or are we trying to correct pathnames that do not point to actual directories (e.g., ACL's PROBE-DIRECTORY)?
My earlier emails confuse these two together. If we are trying to do the former, we can't use the operating system, but if we are trying to do the latter, we pretty much /must/ use the operating system.
I'm inclined to the counsel of despair: the former isn't worth doing, and the latter seems likely to involve both too many cycles, and too much programmer effort to paper over cross-implementation and cross-OS inconsistencies.
Best, R
Richard M Kreuter wrote:
Gary King writes:
OK, I /have/ misdiagnosed this. The logic seems actually busted in directory-pathname-p. The problem is that, at least on allegro, you can get a valid directory pathname whose name component is neither NIL, nor :unspecific, but "" (the empty string).
Ugh,
(member (pathname-name pathname) (list nil "" :unspecific) :test 'equal)
A source of the problem are pathnames that arise from things like this:
> (make-pathname :name "" :directory '(:absolute "tmp")) #p"/tmp/" > (pathname-name *) ""
Added a call to namestring seems to be another way to canonical things so I think this will also work and feels (to me) a bit more portable:
(defun directory-pathname-p (pathname) (let ((pathname (namestring pathname))) (and (member (pathname-name pathname) (list nil :unspecific)) (member (pathname-type pathname) (list nil :unspecific)))))
I see two problems:
(1) If the the host component of *DEFAULT-PATHNAME-DEFAULTS* is different than the host component of PATHNAME, the parse might fail or come out wrong.
(2) Pathnames with "" for the name component don't have namestrings under SBCL. (IIUC, the intent there is to try to have namestring parsing and unparsing be non-lossy.)
I think it might do just as well to say that "" for the name counts the same as NIL or :UNSPECIFIC for DIRECTORY-PATHNAME-P, unless somebody knows of implementations that don't work that way.
The problem with this is that it is open to the kind of attacks that Andreas and Zach propose. Note that those already break the existing directory-pathname-p assumption that one can check the name and type components separately....
[I am pleased to reveal that ACL, at least, throws away name fields of "."... Haven't checked SBCL, nor have I yet checked to see if these are acceptable name components.]
Agreed. If we claim that a directory is a pathname 'without" a filename, then the three ways (so far) for a filename to be missing are for it to be `nil`, the empty string or :unspecific, Here's another crack at it and a sort-of test
(defun directory-pathname-p (pathname) "A directory-pathname is a pathname _without_ a filename. The three ways that the filename components can be missing are for it to be `nil`, `:unspecific` or the empty string." ;; a rather ugly implementation of the docstring (let ((null-components (list nil :unspecific ""))) (flet ((check-one (x) (not (null (member (pathname-name pathname) null-components :test 'equal))))) (and (check-one (pathname-name pathname)) (check-one (pathname-type pathname))))))
#+(or) ;;test (every (lambda (p) (directory-pathname-p p)) (list (make-pathname :name "." :type nil :directory '(:absolute "tmp")) (make-pathname :name "." :type "" :directory '(:absolute "tmp")) (make-pathname :name nil :type "" :directory '(:absolute "tmp")) (make-pathname :name "" :directory '(:absolute "tmp")) (make-pathname :type :unspecific :directory '(:absolute "tmp")) (make-pathname :name :unspecific :directory '(:absolute "tmp")) (make-pathname :name :unspecific :directory '(:absolute "tmp")) ))
On Jul 13, 2009, at 1:57 PM, Richard M Kreuter wrote:
Gary King writes:
OK, I /have/ misdiagnosed this. The logic seems actually busted in directory-pathname-p. The problem is that, at least on allegro, you can get a valid directory pathname whose name component is neither NIL, nor :unspecific, but "" (the empty string).
Ugh,
(member (pathname-name pathname) (list nil "" :unspecific) :test 'equal)
A source of the problem are pathnames that arise from things like this:
(make-pathname :name "" :directory '(:absolute "tmp"))
#p"/tmp/"
(pathname-name *)
""
Added a call to namestring seems to be another way to canonical things so I think this will also work and feels (to me) a bit more portable:
(defun directory-pathname-p (pathname) (let ((pathname (namestring pathname))) (and (member (pathname-name pathname) (list nil :unspecific)) (member (pathname-type pathname) (list nil :unspecific)))))
I see two problems:
(1) If the the host component of *DEFAULT-PATHNAME-DEFAULTS* is different than the host component of PATHNAME, the parse might fail or come out wrong.
(2) Pathnames with "" for the name component don't have namestrings under SBCL. (IIUC, the intent there is to try to have namestring parsing and unparsing be non-lossy.)
I think it might do just as well to say that "" for the name counts the same as NIL or :UNSPECIFIC for DIRECTORY-PATHNAME-P, unless somebody knows of implementations that don't work that way.
-- RmK
-- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
Partial fix for directory-pathname-p.
diff --git a/asdf.lisp b/asdf.lisp index 6a8453b..851b5b5 100644 --- a/asdf.lisp +++ b/asdf.lisp @@ -433,7 +433,10 @@ which evaluates to a pathname. For example: ")
(defun directory-pathname-p (pathname) - (and (member (pathname-name pathname) (list nil :unspecific)) + "Is PATHNAME intended to represent a directory? +Note that this does NOT check to see that PATHNAME points to an +actually-existing directory." + (and (member (pathname-name pathname) (list "" nil :unspecific) :test 'equal) (member (pathname-type pathname) (list nil :unspecific))))
(defun pathname-name+type (pathname)
Hi Robert,
I think our code passed in the ether. (I think that) The only substantive difference between your version and mine is that mine checks both type and name against ("" nil :unspecific) whereas yours treats name and type differently. What's up with that? <smile>
thanks,
On Jul 13, 2009, at 2:55 PM, Robert Goldman wrote:
Partial fix for directory-pathname-p.
diff --git a/asdf.lisp b/asdf.lisp index 6a8453b..851b5b5 100644 --- a/asdf.lisp +++ b/asdf.lisp @@ -433,7 +433,10 @@ which evaluates to a pathname. For example: ")
(defun directory-pathname-p (pathname)
- (and (member (pathname-name pathname) (list nil :unspecific))
- "Is PATHNAME intended to represent a directory?
+Note that this does NOT check to see that PATHNAME points to an +actually-existing directory."
- (and (member (pathname-name pathname) (list ""
nil :unspecific) :test 'equal) (member (pathname-type pathname) (list nil :unspecific))))
(defun pathname-name+type (pathname)
asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
-- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
I haven't really thought about this properly, but why not just:
(defun sysdef-central-registry-search (system) (let ((name (coerce-name system))) (block nil (dolist (dir *central-registry*) (let* ((defaults (probe-file (eval dir))) (file (when defaults (make-pathname :defaults defaults :version :newest :name name :type "asd" :case :local)))) (when (and file (probe-file file)) (return file)))))))
That way, as long as the Lisp and OS agree that the pathname in *CENTRAL-REGISTRY* more or less denotes a directory everything should just work -- assuming that in sane environments (probe-file "/foo/bar") returns something with (:absolute "foo" "bar") in the directory component as long as it actually is a directory.
Cheers,
-- Nikodemus
Nikodemus Siivola wrote:
I haven't really thought about this properly, but why not just:
(defun sysdef-central-registry-search (system) (let ((name (coerce-name system))) (block nil (dolist (dir *central-registry*) (let* ((defaults (probe-file (eval dir))) (file (when defaults (make-pathname :defaults defaults :version :newest :name name :type "asd" :case :local)))) (when (and file (probe-file file)) (return file)))))))
That way, as long as the Lisp and OS agree that the pathname in *CENTRAL-REGISTRY* more or less denotes a directory everything should just work -- assuming that in sane environments (probe-file "/foo/bar") returns something with (:absolute "foo" "bar") in the directory component as long as it actually is a directory.
Cheers,
-- Nikodemus
We got led down this rathole by trying to provide more helpful support to users whose *central-registry* had bad entries in it.
Note, BTW, that what you are proposing runs aground on Allegro, at least. /Users/rpg/lisp on my machine is a directory, but (probe-file "/Users/rpg/lisp" does NOT return a pathname with '(:absolute "Users" "rpg" "lisp"):
ltml(38): (probe-file "/Users/rpg/lisp") #P"/Users/rpg/lisp" ltml(39): :i * A new pathname struct @ #x1017f5d042 = #P"/Users/rpg/lisp" 0 Class --------> #<structure-class pathname> 1 host ---------> The symbol nil 2 device -------> The symbol :unspecific 3 directory ----> (:absolute "Users" "rpg"), a proper list with 3 elements 4 name ---------> A simple-string (4) "lisp" 5 type ---------> The symbol nil 6 version ------> The symbol :unspecific 7 namestring ---> A simple-string (15) "/Users/rpg/lisp" 8 hash ---------> The symbol nil 9 dir-namestring -> The symbol nil 10 plist --------> The symbol nil
This is PRECISELY the sort of case that led to the proposition that started this chain, I believe. The above pathname would have caused a continuable error, warning me that I should have written "/Users/rpg/lisp/" instead of "/Users/rpg/lisp" on ACL....