On 4/18/10 Apr 18 -10:48 AM, Juan Jose Garcia-Ripoll wrote:

On Sun, Apr 18, 2010 at 12:20 AM, Faré <fahree@gmail.com <mailto:fahree@gmail.com>> wrote:

PS: after 2.0 is out, I'd like to resign from ASDF maintainership, to focus on XCVB (help welcome on that, too). Is anyone volunteering to replace me on ASDF?

After thinking about it, I am willing to step up as a possible replacement. The reasons why I would go that far are .... Among my goals would be to implement a faster development model that is based on a more agile dialogue * Feature is proposed with a model implementation * Arguments in pro/cons in the list * Implementation committed only to an asdf-devel branch or to a personal repo. * Verification that all relevant software builds * Iterate from (2) * Repeat until enough features grant a push to master and a release.

For what it's worth, I'm happy to see an agile development, but I would strongly discourage implementation as the main way of introducing new features.

From my experiences working on the ASDF documentation and the innards of TRAVERSE, one of the hardest things about working on ASDF is that there is no clear, written down description of what ASDF /should/ do.

I'm sure it may seem that having a spec might make this less agile, but on more reflection, I believe that the community would see that this is not the case. Right now, the only way to get started working on ASDF is to read a great deal of code, and understand a number of subtleties and oddities about how ASDF functions now. If there was, for example, a clear description of the plan-generation function that is implemented in TRAVERSE, it would be a lot easier for someone to start working on it. I can speak from hard experience! Similarly, now Faré has given it the once-over, but we have no very good way of comparing old and new behaviors and determining which (if any!) is correct, because we have no criteria for what it means for TRAVERSE to be "correct." ASDF is central enough to the entire Lisp community, that it deserves a specification. Perhaps at the ILC or a Euro Lisp Symposium we could have a workshop on ASDF and its future. Best, r

Dear all, I think that Juanjo's and Janderson's propositions are on the right track, and I'm glad Juanjo is stepping forward as ASDF maintainer. I just have a few points to make: * while being the XCVB author and ASDF maintainer, I've become acutely aware of the distance between a simple idea, a prototype that works in the common, a robust product that works in all cases, and a robust documented library that is suitable for developers, and a polished product that is usable by end-users. * I don't think it's demeaning to say that the prototypes developed by Juanjo or James are not polished products yet. I could argue at length on the details, but my main point in not including them so far is that I don't think they are ready for primetime, and I am not willing to either work out their issues or to delay ASDF 2.0 while said issues get worked out. * When Juanjo, Janderson, or someone else takes over after 2.0, it will be time to mainline those ideas, and the new maintainer will have to do the painful work of resolving each and every bug until the code is right. Ouch. * You don't have to wait until after ASDF 2.0 is released to start branches or alternate repositories. I would encourage the creation of such branches. * Be aware that no branch should be merged in until the code is robust enough, usable by its target audience, and has someone willing to fix any issues that come up with testing (and, ideally, includes proper unit tests). Now, as of the development and testing process, * I think Juanjo's investigation of ASDF practice is essential in establishing what "backwards compatibility" is really needed. If we know exactly which packages are affected by some change, we can make that change and fix those packages. * Janderson's automated testing is also a huge step in the right direction. As with the ASDF internal test suite, the goal would be to first establish a baseline what backwards-compatible build success is, and from there be able to detect and fix regressions. As for future developments: * I think the design goals that Juanjo put forward are pretty uncontroversial. * I will happily argue the devil out of the details of the proposed patches, but not in this conversation thread. * Having revamped the internals of TRAVERSE for performance reasons (but without changing its semantics), I can tell: it sucks, and no one can sanely be relying on its precise behavior. Don't be afraid of replacing it. * When I started XCVB, I didn't think that ASDF was salvageable, either technically or politically. Thank all of you who participate on this list for proving me wrong. Regarding XCVB itself: * XCVB already embodies a lot of the principles outlined by Juanjo in this thread and on c.l.l, and what it doesn't have yet it certainly was intended to have eventually. http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/9451a5209... * XCVB notably has declarativeness of the system specification, support for generated files, lisp images, cross-compilation, a compile-and-link model, warning control, etc. * Like ASDF, it is lacking good support for testing, documentation, foreign language compilation, shared libraries, general rules, etc., although allegedly the basic model of XCVB has room for them in a way that isn't true for ASDF. * It is also lacking things that ASDF has, like support for more implementations than SBCL, CCL, CLISP, a good extension mechanism and its actual application to compiling CFFI, Ironclad, a working WITH-COMPILATION-UNIT, etc. * If you manage to salvage ASDF and make it evolve incompatibly towards more declarativeness, XCVB will happily be fully compatible with the new, better ASDF. XCVB might then become "just" an alternate, out-of-image cross-compiling implementation of the ASDF specification. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] In the long run, John Maynard Keynes is dead. — John Perich

Dear Juanjo, I'm sorry that my mail was taken the wrong way. It was only meant to explain why I'm rejecting a lot of ideas *FOR NOW*, i.e. until ASDF 2.0 is out, even though I'd like to see SAME IDEAS take form *AFTER RELEASE* and be merged in. I actually agree with your approach and your propositions, I'm just postponing them until after a 2.0 release. My plan is to freeze the master branch at the end of April, thereupon only accepting bug fixes or trivial enhancements, and hopefully declare 2.0 released before the end of May. I was suggesting that for the impatient, a 2.1 branch be created, that would become the master branch after 2.0 is released (with 2.0 itself becoming a secondary branch). As for XCVB, I think that it would make a good start for an ASDF 3, but that doesn't matter much. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] For pretty much every writer, the big problem isn't piracy, it's obscurity. — Tim O'Reilly, as cited by Cory Doctorow On 20 April 2010 04:34, Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com> wrote:

Dear Fare,

I can not avoid to say that much of your previous email needed not be written. * THe warning about incomplete and bogus patches being submitted * The warning that the current maintainer knows it better the pain of maintaining and pushing new features. * Dooms about maintainers having to suffer the consequences of wrong choices right now and how bugs will pop up was unnecessary. * Equally so the warning that XCVB will gladly host all the lispers expelled from ASDF's imcompatible and disruptive changes.

Why you could have saved that

* First of all, the patches I submitted were never claimed to be final. I sent them because all i got was a no, no, bogus design, impossible, useless. I just wanted to show things can be done and can be used. This includes the extensions to OPERATE and TEST-OP to gather the lists of errors and make them really useful, the extension to include logical pathnames, the :TESTS annotation, the last extensions to make defsystem forms have dependencies... However this statement shows why this dialogue I attempted is futile: "I am not willing to either work out their issues or to delay ASDF 2.0 while said issues  get worked out."

* Even if this a mailing list with a relatively low traffic, warning people about our desire to make ASDF behave incompatibly, or our incompetence to prevent bugs that will propagate to later releases is at the very least offensive. The first because it is wrong, the second because you yourself have introduced bugs and fixed them.

* Encouraging *forks* of ASDF is just ungrateful to the project and only shows a, perhasp subconscious, will to weaken it.

Let me get this straight.

* I never spoke about making ASDF behave incompatibly in a disruptive way. This has been said by other people, here, at the ECL mailing list and at c.l.l. but I have always spoken against it.

* I have always been talking about providing tools (i.e. annotations) that allow people evolve into a more declarative syntax, freezing it as the only one when and only when all the relevant packages get it right (99%? ASDF monitoring? These are and have been my words.)

Juanjo

-- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

On Tue, Apr 20, 2010 at 3:39 PM, Faré <fahree@gmail.com> wrote:

I'm sorry that my mail was taken the wrong way.

I understand that email is not always the best medium for communicating.

My plan is to freeze the master branch at the end of April, thereupon only accepting bug fixes or trivial enhancements, and hopefully declare 2.0 released before the end of May.

I think this is the right thing to do given the scarcity of resources. That would also give us time to set up the infrastructure I mentioned w.r.t. testing and measuring the influence of ASDF's design choices. It would also help in creating a wiki (anybody opposes it living in wikispaces and doing it _now_?) To name one thing, not all the features present in ASDF 2 are actually spread and used throughout the Common Lisp world. To name one: the :type argument to components which allows specifying the file extension. There are many asd files out there that define classes just for that. It would make sense to write down a document that lists the key changes and how they should be used (pointer to the manual, etc) Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

On 2010-04-21, at 04:00 , Faré wrote:

Dear James,

...

before one can declare asdf 2.0 to be feature complete, one should have a specification of its intended function and then gauge the implementation in those terms.

I humbly disagree. The standards that I have set to myself for ASDF 2.0 are as follows:

[ ... which list of goals should well appear in release notes, but does not address the concern raised in this thread, except in the exclusion, that ... ]

EXPLICITLY NOT INCLUDED amongst my goals for ASDF 2.0, are the following goals that a future maintainer may or may not include as his goal for a future release:

* [ ... ] * keep ASDF under any specific code size.

were one to be in a position to consider responsibility for the future maintenance of the code, that item would appear quite as far down on the list.

...

while the users' manual does explain aspects of the user and/or programmer interface, in addition to the descriptions of the interface, there should be something which describes the requirements which are being beta-tested - both to assess completeness and to inform future maintenance.

This is too abstract for me. If there are specific elements you think should be documented, please list them concretely.

i will help you. there is a section in the asdf manual called "Controlling where ASDF saves compiled files". that section describes how a user can configure asdf to translate output files. there should be two additional sections in a manual, perhaps a developers' manual, called "output translation requirements" and "output translation implementation" which describe - independent of how to use the facility, what it does and how it is implemented. a "requirements" section should describe at least - what kinds of output translations are required. it should do this in terms of abstract characteristics, specific use cases, and enumerative examples; - the operating systems and the project forms to be supported by asdf with attention to the required performance; - what trade-offs are apparent in supporting which functional features, which options were elected, and why; - an abstract model for the entities which the implementation is expected to represent, support, manipulate. an "implementation" section should provide an overview of the one hundred twenty functions and twenty parameters which have been introduced in since version 1.362, with attention to how their relation to the functional requirements. section "8.6" in the asdf manual reads as if it might have been intended as notes for a section of this sort. if this is still too abstract, any one of the readily available descriptions of "functional requirements", eg, here[1], might may help you to understand their purpose and construction.

One way you could do it is to send a patch of a first draft of a modification of the manual that you require, with holes where you want me to fill information.

...

the additions to the code base over the past few months were devoted to two areas:

- configuration - output translations

i neglect, for the moment, the configuration component of the asdf evolution and concentrate on questions related to output translations. even though the configuration mechanism contributes settings which affected the latter, they are in sufficiently independent, that an isolated analysis is reasonable.

Well, most of the translation code *IS* the configuration system. The output-files :around method itself is small, and its helpers are relatively simple, except for the ugly directorize-pathname-host- device required for portability (that none of the previous solutions provided).

...

in lieu of a specification, several asdf-devel messages on the topic of output translations[1--5] discuss requirements. the asdf-binary- location users' guide includes a "what it does" passage[6]. the asdf- binary-locations-devel archives[7] include no additional discussions. to follow those discussions, an output translation mechanism should make it possible to

- place (output) files under a single root - place (output) files in a tree specific to (runtime x version x project x file-type) - include process information (eg, pid or user id) in the location - specify elements of the mapping programmatically - specify elements of the mapping by configuration - combine independent specifications - express location maps both globally and in connection with specific sub-systems (cl-controller, cl-launch, etc) - support simultaneous, independent mapping for binary output and ancillary data files - disable translations for a given path - disable translations completely

is this list complete? if not, it would help if you would complete it. it or something equivalent should appear in a "requirements" section.

...

in response to these requirements, "asdf binary locations" was developed, first as an extension to, then integrated with asdf, and further developed into a general output translation facility. while the evolution brings definite benefits, in the process, between version 1.362, when a-b-l was incorporated into asdf core, and the current +/- 1.7, the source size has increased from 56K to 132K, the function count has grown from less than one hundred to more than two hundred, and the call count has almost tripled.

Yes. Configuration support has doubled the size of ASDF, and added features, cleanups and robustification have also increased its size. I don't think any of this stuff can be done without.

...

the call graphs[8] depict the situation. each node in the graph corresponds to a function in the asdf implementation, each edge to a call. the green nodes were present in 1.362, the yellow nodes in 1.502, and red nodes in 1.7. filled nodes have remained, while empty nodes indicate functions which are no longer present. the increase in code mass and complexity is evident. since, as expressed, the requirements could be met by logical pathname translations, the situation causes concern. there is nothing in the functional requirements, as such, which dictates an implementation of this complexity.

I don't think the implementation is particularly complex. It is larger because it does more things.

Make things as simple as possible BUT NO SIMPLER.

Show me which function you'd like to see gotten rid of, and I'll explain you why it is necessary - or get rid of it.

the patch which was enclosed in the earlier message[2] demonstrated an implementation which modifies the three functions - component- pathname, input-files, and output-files, extends the function which constructs the component absolute pathname, and adds a function to map between logical and physical pathname components. with respective calls from the other five functions. without having deconstructed the asdf 1.7* implementation to distinguish which functions relate to which functional components, on first order, five is five percent one hundred and twenty and an even smaller percentage of two hundred.

...

there are, however, non-functional concerns which appear to have informed the decision: it has been held, that - logical pathnames are not portable; - the facility must operate in terms of physical pathnames as real- world lisp applications require file names which do not conform to logical pathname syntax constraints.

Yes.

...

in the course of testing the output location mapping itself, it was demonstrated[9] that the non-portability claims were unfounded. an experiment based on 1.502 demonstrated[10] that a mechanism which separates location translation from truename processing can provide the required functionality with much less code in terms of logical pathnames despite non-conformant physical pathnames. the logical pathname based location maps were then re-implemented in 1.702 and further experiments demonstrated that - with changes to the scripts to account for functions which would be removed, all regression tests pass, and that with appropriate initialization, a multi-runtime build is supported for the normal abcl/acl/ccl/clisp/cmucl/ecl/lw/sbcl complement. the following host definition, for example, produces a centralized per-runtime translation tree for output files.

(let ((root (make-pathname :directory (butlast (pathname-directory *build-init-pathname*) 2) :name nil :type nil :defaults *build-init- pathname*)) (host "SOURCES")) (setf (logical-pathname-translations host) '(("**;*.*" "**;*.*"))) (let ((lbin (pathname-type (compile-file-pathname (make- pathname :host host :name "TEST" :type "LISP")))) (bin (pathname-type (compile-file-pathname "test.lisp")))) (setf (logical-pathname-translations host) `((,(make-pathname :host host :directory '(:absolute :wild- inferiors) :name :wild :type lbin :version :wild) ,(make-pathname :directory `(,@(pathname-directory root) "bin" ,(asdf::implementation-identifier) :wild-inferiors) :name :wild :type bin :version :wild)) (,(make-pathname :host host :directory '(:absolute :wild- inferiors) :name :wild :type :wild :version :wild) ,(make-pathname :directory `(,@(pathname-directory root) :wild-inferiors) :name :wild :type :wild :version :wild))) ))

(setq asdf::*translations-root* (make-pathname :host host :directory '(:absolute) :name nil :type nil)))

given which, the question occurs, whether the output locations implementation deserves further thought before it advances to beta status?

Less code that what? How is the end-user meant to deal with configuration? The horribly complex logical pathname translation above is not what I would like to have to explain to users in the manual.

the irony in a situation, where a doubled code size and tripled call count are insignificant while a two-clause logical host definition is incomprehensible, is appreciable, but not significant. the matter at issue is not whether you can read logical host specifications, but whether the attendant translation mechanism would simplify asdf implementation.

If you want people to adopt logical pathnames -- get the spec fixed, and convince implementers (notably SBCL implementers) to adopt your fixed spec.

we can discuss the general utility of logical pathnames at length, at your leisure. elsewhere. my enquiry concerns whether logical pathname translations would serve as a more compact implementation mechanism for asdf output translations than the present method.

And even then you'll have a hard time fixing things that users can now trivially do with the following translation:

(:root ("C:/my/cl/cache" :implementation))

This will work and make his file in \\remote-host\myshare\cl\cl-foo\foo-V2.000\bar_baz.lisp be compiled as obvious from the spec into C:\my\cl\cache\sbcl-1.0.38-windows-x86\remote-host\myshare\cl\cl- foo\foo-V2.000\bar_baz.fasl

this would appear to be the equivalent of a translation on the order of `("**;*.*.* ,(concatenate 'string "C:/my/cl/cache/" (asdf::implementation-identifier) "/**/*.*.*")) but i may have misunderstood the requirement. --- [1] : http://en.wikipedia.org/wiki/Functional_requirement [2] : http://common-lisp.net/pipermail/asdf-devel/2010-April/001351.html

On 2010-04-21, at 10:56 , Juan Jose Garcia-Ripoll wrote:

On Wed, Apr 21, 2010 at 10:31 AM, james anderson <james.anderson@setf.de> wrote: the patch which was enclosed in the earlier message[2] demonstrated an implementation which modifies the three functions - component- pathname, input-files, and output-files, extends the function which constructs the component absolute pathname, and adds a function to map between logical and physical pathname components. [...] we can discuss the general utility of logical pathnames at length, at your leisure. elsewhere. my enquiry concerns whether logical pathname translations would serve as a more compact implementation mechanism for asdf output translations than the present method.

...

And even then you'll have a hard time fixing things that users can now trivially do with the following translation:

(:root ("C:/my/cl/cache" :implementation))

This will work and make his file in \\remote-host\myshare\cl\cl-foo\foo-V2.000\bar_baz.lisp be compiled as obvious from the spec into C:\my\cl\cache\sbcl-1.0.38-windows-x86\remote-host\myshare\cl\cl- foo\foo-V2.000\bar_baz.fasl

this would appear to be the equivalent of a translation on the order of

`("**;*.*.* ,(concatenate 'string "C:/my/cl/cache/" (asdf::implementation-identifier) "/**/*.*.*"))

Let's try to clarify the discussion.

* Fare insists on the validity of the current pathname translation mechanism, which allows the user to specify the destination of compiled files using physical pathnames and some configuration file or S-expr. If the user knows where the original file lived, he can find where the compiled file lives easily -- one to one mapping, no ambiguity, no names changes except mapping everything to a new tree.

* James is claiming that this can be implemented using _internally_ logical pathnames with a logical pathname tree structure that is constructed from the component names and not from the physical location of the files they represent. This would be complemented with a function to obtain the actual location of compiled files given the component object. The configuration can be mapped to this structure and the overall implementation would then be smaller.

in principle, yes. in detail, there is no need to construct a global structure as the computation can proceed locally.

I see shortcomings on both sides

- Code size - A bunch of people already using asdf-binary-locations out there (which is what was merged in)

which one should be able to continue to support in any case.

- James' implementation only seems to allow one file per component and it does not seem to take into account when OUTPUT-FILES returns more than one file.

to quote from the original post.

further experiments demonstrated that - with changes to the scripts to account for functions which would be removed, all regression tests pass, and that with appropriate initialization, a multi-runtime build is supported for the normal abcl/acl/ccl/clisp/cmucl/ecl/lw/sbcl complement.

that is, the multi-runtime build included ecl. for which the expression of the functional requirements was the conditionalization in the output-files method. if that is the extent of the requirement, then ecl is taken into account.

- In James's model the interrogation: where does the compiled file associated to "file1.lsp" does not make sense without traversing the ASDF system and locating that file and its associated component.

if the intended question was: "does the compiled file associated with 'file1.lsp' make sense without traversing the ASDF system?" yes and no. yes, in that the output - and _input_ - location translations are computed with respect to the respective component only. no, in that system instantiation protocol includes a step which (effectively) walks the model to compute a location for each component. as i noted in the original message, this makes it a lot easier to understand a component pathname computation, but is independent of any translation mechanism. in addition, it is no longer even unique to the lp-based location translation implementation. as the step was already introduced in 1.676.

On 2010-04-21, at 12:12 , Juan Jose Garcia-Ripoll wrote:

On Wed, Apr 21, 2010 at 11:50 AM, james anderson <james.anderson@setf.de> wrote:

...

further experiments demonstrated that - with changes to the scripts to account for functions which would be removed, all regression tests pass, and that with appropriate initialization, a multi-runtime build is supported for the normal abcl/acl/ccl/clisp/cmucl/ecl/lw/sbcl complement.

that is, the multi-runtime build included ecl. for which the expression of the functional requirements was the conditionalization in the output-files method. if that is the extent of the requirement, then ecl is taken into account.

I presume those tests only use the _default_ build system, not ECL's extensions. Your patches were produced against 1.5 and there the OUTPUT-FILES would only return one file, even for ECL.

the build script performed a simple load-system. while the patches are from the first experiment, which was based on 1.5 as it is simpler, the later tests were based on 1.702. i quote, again, from the original message:

[ ... ] the logical pathname based location maps were then re-implemented in 1.702 and further experiments demonstrated that - with changes to the scripts to account for functions which would be removed, all regression tests pass, and that with appropriate initialization, a multi-runtime build is supported for the normal abcl/acl/ccl/clisp/cmucl/ecl/lw/sbcl complement. the following host definition, for example, produces a centralized per-runtime translation tree for output files.

it used a standard project build script for cross-runtime tests.[1,2] as this thread is to discuss in principle, yet another diff would distract. if it would help to have a diff against 1.702, i can get one and post it.

However, once we integrated ECL's extensions, these produce two object files, and I fail to see from your patch how the output files are handled -- from what I can see, since the OUTPUT-FILES method does not have an :AROUND, the output of ECL's extensions is not translated: they just take the source file produce a compile- file pathname and that's it. In other words: it is going to work, but it is going to have files laying around, both translated and not.

i do not understand "laying around". subsequent to the build, the source directory contained source files only and the ecl-.... shadow tree contained .o and .fas files. this is as i had thought it was supposed to have been.

I suppose in your model developers are expected to integrate an explicit call to component-translate-pathname in every method they define for output-files.

yes, as it was implemented that would be the case. the point of the experiment was not to demonstrate the best integration and api, but to investigate whether the mechanism satisfies the functional requirements.

no, in that system instantiation protocol includes a step which (effectively) walks the model to compute a location for each component.

Indeed, I find this an interesting approach, independently of what method is used for translation, but I see a problem if a user decides to temporarily deactivate translations or change the destination of a system.

the implementation intended to leave logical pathnames alone. if the system is expressed in terms of lp-conformant components, there is no reason for asdf to translate anything. it is certainly possible that the patch would not work that way, but it should.

Right now it amounts to changing the translation table. With your model it involves re-reading and rebuilding the system and components.

it should be sufficient to redefine the logical host(s). --- [1] : http://174.129.66.148:8000/metadata/net/common-lisp/alexandria/ build-system.sh [2] : http://174.129.66.148:8000/metadata/lib/html/ build-20100421T111506Z00.html

On 2010-04-21, at 04:00 , Faré wrote:

[...]

...

the additions to the code base over the past few months were devoted to two areas:

- configuration - output translations

i neglect, for the moment, the configuration component of the asdf evolution and concentrate on questions related to output translations. even though the configuration mechanism contributes settings which affected the latter, they are in sufficiently independent, that an isolated analysis is reasonable.

Well, most of the translation code *IS* the configuration system. The output-files :around method itself is small, and its helpers are relatively simple,

the call graph[1] generator counted 66 functions. they appear to be split 1/3 - 2/3 between configuration and computation. that leaves about 44, which is about fifty percent of the pre 1.3 asdf function count.

except for the ugly directorize-pathname-host-device required for portability (that none of the previous solutions provided).

what requirement does that support? in particular, it would help to record in the manual section on "output translation implementation" why the process translates twice and what the respective realms mean. --- [1] : reports.setf.de/asdf-calls-dot.pdf [2] : http://tinyurl.com/y6jv5mb