Would anyone object to augmenting the docstring for COMPONENT-RELATIVE-PATHNAME?

Currently it's:

"Extracts the relative pathname applicable for a particular component."

How about something like:

"Returns a pathname for the component argument relative to the pathname of that component's parent."

?

I think the most important thing for the reader to know is what the return value is relative to.

The above isn't ideal, either, because sometimes what this returns is NOT a relative pathname (at least as far as I can tell by reading the source) --- it extracts the value that the system definer sets using :pathname so that the "relative" pathname may well be absolute!

Perhaps:

"Returns a pathname for the component argument intended to be interpreted relative to the pathname of that component's parent. Despite the function's name, the return value may be an absolute pathname, because an absolute pathname may be interpreted relative to another pathname in a degenerate way."

A little long-winded. I'm open to suggestions for improvement.

What about adding a docstring for SPLIT-PATH-STRING, too --- any function that returns three values is worth a docstring!

"Splits the path string S, returning three values: A flag that is either :absolute or :relative, indicating how the rest of the values are to be interpreted. A directory path --- a list of strings, suitable for use with MAKE-PATHNAME when prepended with the flag value. A filename with type extension, possibly NIL in the case of a directory pathname. FORCE-DIRECTORY forces S to be interpreted as a directory pathname (third return value will be NIL, final component of S will be treated as part of the directory path."

By the way, is there any danger that SPLIT-PATH-STRING will be called on a logical pathname string? I don't see anything to keep it from happening. If so, it looks like it will lose badly; split-path-string assumes that the pathname separator is "/".

If you all agree with me, this looks like a job for a launchpad ticket.

Cheers, r

On 2/19/10 Feb 19 -1:10 PM, Faré wrote:

On 19 February 2010 10:50, Robert Goldman rpgoldman@sift.info wrote:

...

Would anyone object to augmenting the docstring for COMPONENT-RELATIVE-PATHNAME?

Currently it's:

"Extracts the relative pathname applicable for a particular component."

How about something like:

"Returns a pathname for the component argument relative to the pathname of that component's parent."

?

I think the most important thing for the reader to know is what the return value is relative to.

The above isn't ideal, either, because sometimes what this returns is NOT a relative pathname (at least as far as I can tell by reading the source) --- it extracts the value that the system definer sets using :pathname so that the "relative" pathname may well be absolute!

Perhaps:

"Returns a pathname for the component argument intended to be interpreted relative to the pathname of that component's parent. Despite the function's name, the return value may be an absolute pathname, because an absolute pathname may be interpreted relative to another pathname in a degenerate way."

A little long-winded. I'm open to suggestions for improvement.

What about adding a docstring for SPLIT-PATH-STRING, too --- any function that returns three values is worth a docstring!

"Splits the path string S, returning three values: A flag that is either :absolute or :relative, indicating how the rest of the values are to be interpreted. A directory path --- a list of strings, suitable for use with MAKE-PATHNAME when prepended with the flag value. A filename with type extension, possibly NIL in the case of a directory pathname. FORCE-DIRECTORY forces S to be interpreted as a directory pathname (third return value will be NIL, final component of S will be treated as part of the directory path."

By the way, is there any danger that SPLIT-PATH-STRING will be called on a logical pathname string? I don't see anything to keep it from happening. If so, it looks like it will lose badly; split-path-string assumes that the pathname separator is "/".

Split path string will be called on a what is provided as :name to ASDF. In these circumstances, we get to define what is the pathname separator convention (if any) and I think that having one is better than having none, and "/" is a pretty unsurprising one. Yes, that needs to be documented.

...

If you all agree with me, this looks like a job for a launchpad ticket.

No! This looks like a job for someone willing to write the damn doc strings and get their patch committed. Ahem, don't you have commit priviledge?

Sorry, not clear! Two tasks:

1. Get docstrings in. I can do that. I've written them already, just thought I'd get a review --- once Xophe went over one of my docstrings for SBCL with a fine-tooth comb and the results were much better.

I will take care of the docstrings, as long as no one thinks these are bad.

2. SPLIT-PATH-STRING --- this is the one I think might need a ticket. I confess I'm bamboozled by this one. It's called on (component-name component), not on a pathname. Can you explain why the COMPONENT-NAME would end up being a string that looks like a pathname?

This SPLIT-PATH-STRING stuff looks like a relatively recent gwking addition (if I understand the output of git blame correctly). I don't see what use case it was introduced for. Anyone else have a clue? I see this docstring for component-name:

"Component name: designator for a string composed of portable pathname characters"

but I have never seen at component-name that was anything other than a simple name, never something with directory separators. Is this ever used in the wild? Or was this something that was added in an excess of enthusiasm and that should just be killed (like the undocumented tripartite FEATURE form)?

thanks, r

On 2/19/10 Feb 19 -2:19 PM, Faré wrote:

...

2. SPLIT-PATH-STRING --- this is the one I think might need a ticket.

I confess I'm bamboozled by this one. It's called on (component-name component), not on a pathname. Can you explain why the COMPONENT-NAME would end up being a string that looks like a pathname?

This SPLIT-PATH-STRING stuff looks like a relatively recent gwking addition (if I understand the output of git blame correctly). I don't see what use case it was introduced for. Anyone else have a clue? I see this docstring for component-name:

gwking might have done the checkin, but I am the one who wrote the code at ITA, where we wanted to be able to use "foo/bar" as the name of a component, instead of having all those ugly :pathname #.(make-pathname ...) ALL OVER THE DAMN PLACE for hundreds of components.

...

"Component name: designator for a string composed of portable pathname characters"

but I have never seen at component-name that was anything other than a simple name, never something with directory separators. Is this ever used in the wild? Or was this something that was added in an excess of enthusiasm and that should just be killed (like the undocumented tripartite FEATURE form)?

Yes, we use it a lot at ITA, and ASDF-DEPENDENCY-GROVEL and XCVB's ASDF front-end and backend use it, too.

This change is 100% backwards compatible, since the behavior before then was "undefined" when the #/ character appeared in a name. Now it's defined and portable.

OK, so it's defined and portable. Great. Now let's get it documented, and it will be great!

A good way to dodge the (until now) busted module dependencies, too! ;-)

I'll try to get those docstrings committed, and maybe add one for component-name.

r

a question:

On 2010-02-22, at 00:46 , Robert Goldman wrote:

On 2/19/10 Feb 19 -2:19 PM, Faré wrote:

...

...

2. SPLIT-PATH-STRING --- this is the one I think might need a

ticket. I confess I'm bamboozled by this one. It's called on (component- name component), not on a pathname. Can you explain why the COMPONENT- NAME would end up being a string that looks like a pathname?

This SPLIT-PATH-STRING stuff looks like a relatively recent gwking addition (if I understand the output of git blame correctly). I don't see what use case it was introduced for. Anyone else have a clue? I see this docstring for component-name:

gwking might have done the checkin, but I am the one who wrote the code at ITA, where we wanted to be able to use "foo/bar" as the name of a component, instead of having all those ugly :pathname #.(make-pathname ...) ALL OVER THE DAMN PLACE for hundreds of components.

...

"Component name: designator for a string composed of portable pathname characters"

but I have never seen at component-name that was anything other than a simple name, never something with directory separators. Is this ever used in the wild? Or was this something that was added in an excess of enthusiasm and that should just be killed (like the undocumented tripartite FEATURE form)?

Yes, we use it a lot at ITA, and ASDF-DEPENDENCY-GROVEL and XCVB's ASDF front-end and backend use it, too.

This change is 100% backwards compatible, since the behavior before then was "undefined" when the #/ character appeared in a name. Now it's defined and portable.

OK, I'm interested in writing up the documentation, and have been thinking about how this is to be treated. Some thoughts:

1. SPLIT-PATH-STRING is not an ideal name. SPLIT-PATH-STRING is /

not/ used on paths, it is used on /names/ (in the ASDF sense of these terms). An unwary reader of this code might try to apply it to a pathname (as I originally thought), where it could cause lossage because S-P-S doesn't handle logical pathnames (or lots of other things about pathnames). Suggest that we rename this to something like NAME-TO-PATHNAME or NAME-PATHNAME-SPLIT.

2. We should define in the grammar simple-names and structured names.

Simple names have no "/" and structured names may have a slash.

3. The period character should be forbidden in /both/ simple and

structured names because of oddities with pathname types.

4. Only simple names should be permitted in naming systems.

5. For modules, I propose either:

   5.1 Only simple names should be permitted in naming modules, or

   5.2 The use of a structured name implicitly overrides the default

relative pathname for modules. [a test should go here!]

On a related note, the grammar in the manual seems to suggest that a component can be just a string, which I don't believe that ASDF supports.

The grammar says:

component-def := simple-component-name | ( component-type name {option}* )

But I don't believe SIMPLE-COMPONENT-NAME is actually supported. I propose to squash this from the documentation, unless corrected.

why is this better than to leave names atomic and provide a standard syntax to parse component relative (sic) pathnames?

...

why is this better than to leave names atomic and provide a standard syntax to parse component relative (sic) pathnames?

Note that my whole last email is a red herring wrt this question.  My last email assumes that Fare's change stays in, and I'm trying to write it up in the documentation.

I.e., "let's you and him fight."  I am bowing out of this discussion.

I don't understand the question and the proposed alternatives, so I can't argue an answer.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] If you waste time explaining why a socialist's welfare plan will fail, it didn't: it successfully diverted your attention from a Power Grab.

On 2/22/10 Feb 22 -3:37 AM, james anderson wrote:

good morning;

On 2010-02-22, at 01:41 , Robert Goldman wrote:

...

On 2/21/10 Feb 21 -6:35 PM, james anderson wrote:

...

a question:

...

...

[...]

...

why is this better than to leave names atomic and provide a standard syntax to parse component relative (sic) pathnames?

Note that my whole last email is a red herring wrt this question. My last email assumes that Fare's change stays in, and I'm trying to write it up in the documentation.

I.e., "let's you and him fight." I am bowing out of this discussion.

i assert no disagreement here. i read a note, which opened with the confession, that these bits "bamboozle", and then proceeded over a page to attempt to construct a logic which would remain stable in the reader's mind. in light of which, i express the concern, should someone relatively experienced with this code need to go to these lengths, is it the right thing to have done?

I am inclined to agree. I'd be happier if we could just say something like

(:file "foo" :relative-directory "bar")

instead of

(:file "bar/foo")

For that matter, both of these are probably unnecessary now that module dependencies work and we can do:

(:module "bar" :components ((:file "foo")))

So, question: Fare, do we really need the additional hair now that intra-system module dependencies work? Wouldn't it be possible for you to macrologize your names-as-pathnames extension and purge the hair from the core of ASDF?

I'd love to see all this extra code get purged....

cheers, r

On 2/22/10 Feb 22 -10:02 AM, Faré wrote:

...

I am inclined to agree. I'd be happier if we could just say something like

(:file "foo" :relative-directory "bar")

instead of

(:file "bar/foo")

Why? You're just moving the complexity around, without simplifying the overall design. Moreover, the astute user is already familiar with /-separated paths (from URLs if not from his filesystem) and the second notation leverages this familiarity. Which comes very handy when files are shuffled around. Finally, we need to parse names if only to portably handle the :static-file "foo.lisp" case -- why can't we parse paths at the same time?

Least surprise: reuse familiarity with filesystem paths and/or URLs Conciseness: fewer slots that get out of sync Generality: share the same mechanism with :static-file "foo.lisp"

Counterargument:

Confusion/duplication: Compute the component-relative-pathname through some confusing combination of :pathname and :name. Abuse of the :name to provide pathname functionality.

Non-orthogonality: Compute the pathnames of modules and files differently (name not used to give

...

For that matter, both of these are probably unnecessary now that module dependencies work and we can do:

(:module "bar" :components ((:file "foo")))

Not always doable. Especially when dependencies change and suddenly subdirectories are no longer perfect units of dependency. Been there, done that.

You don't have to map modules one to one to files. E.g.

(:module "bar" :components ((:file "foo"))) (:module "bar1" :pathname "bar/" :components ((:file "bletch"))

Right?

I have a project here with over 1400 files, not counting libraries. That trick doesn't work.

...

So, question: Fare, do we really need the additional hair now that intra-system module dependencies work? Wouldn't it be possible for you to macrologize your names-as-pathnames extension and purge the hair from the core of ASDF?

The hair is in the notion of CL pathnames itself. Get rid of it. How do you think this feature can be modularized anyway?

I don't agree. I think the hair comes from putting pathname computation into two places: into the name and into the :pathname. I'd rather see all the pathname computation delegated to the :pathname argument.

...

I'd love to see all this extra code get purged....

I disagree. If you compare with the *broken* mess that was before and led to non-portable behavior in :static-file "foo.lisp", then no, this is not extra code. This is *less* code than was before, for more functionality, and in a more portable way.

The feature is now solved. It is a bug in the documentation if it says otherwise.

Actually, now that I think of it, for portability reasons, and to avoid all these ugly :pathname #.(make-pathname ...) any string argument to :pathname should be processed through component-name-to-pathname-components. You can still #p"..." if you want your system's non-portable parse-namestring, or #.(make-pathname ...) if you are crazy about it.

Currently, :pathname <string> is non-portable and thus USELESS.

I'd rather fix :pathname than add hair to name....

Cheers, r

1. SPLIT-PATH-STRING is not an ideal name.  SPLIT-PATH-STRING is /not/

used on paths, it is used on /names/ (in the ASDF sense of these terms).  An unwary reader of this code might try to apply it to a pathname (as I originally thought), where it could cause lossage because S-P-S doesn't handle logical pathnames (or lots of other things about pathnames). Suggest that we rename this to something like NAME-TO-PATHNAME or NAME-PATHNAME-SPLIT.

What about COMPONENT-NAME-TO-PATHNAME-COMPONENTS or SPLIT-COMPONENT-NAME-INTO-PATHNAME-COMPONENTS

2. We should define in the grammar simple-names and structured names.

Simple names have no "/" and structured names may have a slash.

Where do we care? System names are not passed to the above function.

3. The period character should be forbidden in /both/ simple and

structured names because of oddities with pathname types.

Wait, there are plenty of (:static-file "foo.bar") around the world! Do you propose to make them invalid?

4. Only simple names should be permitted in naming systems.

Why? System names are never parsed into paths.

5. For modules, I propose either:

5.1 Only simple names should be permitted in naming modules, or

It's often useful to name modules that are several levels deep. We used to have modules with :pathname #.(make-pathname :directory '(:relative "foo" "bar")) and it ain't pretty.

5.2 The use of a structured name implicitly overrides the default relative pathname for modules.  [a test should go here!]

Isn't that what's currently happening? If not, that's a bug that should be fixed and tested for. Sigh. Too little testing happening indeed.

What about we use Stefil for testing ASDF?

On a related note, the grammar in the manual seems to suggest that a component can be just a string, which I don't believe that ASDF supports.

The grammar says:

component-def  := simple-component-name                | ( component-type name {option}* )

But I don't believe SIMPLE-COMPONENT-NAME is actually supported.  I propose to squash this from the documentation, unless corrected.

If not currently supported, it's obviously not used anywhere, and can be removed in a backward-compatible way.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The revolution will be won, not, as the authoritarians would claim, when the last political boss is hanged with the bowels of the last state propagandist, but, as the libertarians know, when the ever-repeated attempts to lure people into subservience are systematically greeted with laughter and drowned in ridicule by free men who are well-armed both physically and intellectually.

...

...

2. We should define in the grammar simple-names and structured names.

Simple names have no "/" and structured names may have a slash.

Where do we care? System names are not passed to the above function.

We care for two reasons:

1. To put in the DEFSYSTEM grammar in the manual and

2. So that people don't later on try to hijack this function to do

something for which it was not intended.

Can't we just say that a component name is a string, then explain how the pathname default is computed from that string?

...

...

3. The period character should be forbidden in /both/ simple and

structured names because of oddities with pathname types.

Wait, there are plenty of (:static-file "foo.bar") around the world! Do you propose to make them invalid?

Ick.  Good point.  OK, have to leave those.  Note that we have already documented that those can lead to oddities across lisps.

At ITA we also have directories named after protocol versions, i.e. "foo-V1.200". We do want to keep the ability to have such names, thank you.

...

...

5.2 The use of a structured name implicitly overrides the default relative pathname for modules.  [a test should go here!]

Isn't that what's currently happening? If not, that's a bug that should be fixed and tested for. Sigh. Too little testing happening indeed.

It's not that it isn't happening, it's that we need to specify what /should/ happen in the manual, for the next time someone wades in and starts changing things....

Yes. Documenting and testing are a bitch. They are what is preventing a 2.0 release now.

...

What about we use Stefil for testing ASDF?

I'm a little reluctant to see this done, if only because I don't want to force either additional training, or additional dependencies on the few people who bother to submit patches.

This will make STEFIL the third unit test framework I will have to master (NST, our own; FiveAM for the JSON library); where does this stop?

I'm fine if you think either NST or FiveAM fits the bill, as long as the thing is free software and is supported by clbuild. I think STEFIL is simple enough to learn and use.

And does STEFIL handle the kinds of tests we have, where you have to start new lisp jobs to do tests (seems like many test frameworks don't).

We may have to bind special variables (e.g. *defined-systems*) and to delete-package stuff created by test files, possibly using fixtures. I suppose any half-decent testing infrastructure can do it. I like the general design of STEFIL, but I will accept about anything.

That said, I don't care enough to say we should /not/ do this if someone else has the energy to do it.

If no one does anything nor raises a valid objection, I may commit something someday. Though I'd be more interested in revamping XCVB.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] "If god doesn't like the way I live, Let him tell me, not you." — As seen on a button

On 2010-02-19, at 21:47 , Faré wrote:

On 19 February 2010 06:49, james anderson james.anderson@setf.de wrote:

...

this problem has been in asdf "forever". i have always just patched it locally, but as i've now thrown a few things in the net which other folks should be able to build, i suggested the correction.

Thanks for the correction.

...

it might make sense to consider an entirely different protocol for constructing the component pathnames. in order to better predict the result, asdf-x tries to turn the protocol around and do it from the top down, rather always on-demand from the bottom up. the bottom-up protocol has to recurse to the root anyway in order to construct the intended pathnames.

The problem with an entirely different protocol is compatibility. A change would break things for some people. Who? Where? I don't know. It would break things for me, and I'm willing to deal with it,

i'm curious what would break if the effect of the protocol change was that component pathname always actually returned pathnames which conformed to any reasonable spec for which the present implementation could possibly have been written. what kinds of things can one expect from it except relative pathname extension (modulo '..') and absolute rerooting. the rfcs on uri resolution are detailed enough that it's a known world. that the present implementation does different things in different runtimes is enough to mark anything which depended on its behavior as non-portable and deprecated.

does any other issue which come besides the consequence of modifying the graph to chance module/system constituency. in which case, upon reflection, one would need to automatically re-root the effected components anyway.

but would it break things for someone, somewhere who will be pissed off? Maybe, maybe not. Who's to take the risk?

maybe asdf needs an rfc process.

If you're into designing better protocols for better tools, why not jump on the XCVB bandwagon where you don't have anyone to be incompatible with, and plenty of room to do something useful instead of language lawyering?

because the core problem which asdf should be able to solve is really a rather simple one, as things go, which it, itself should be made to solve better.