On Sun, Apr 18, 2010 at 4:56 PM, Robert Goldman rpgoldman@sift.info wrote:

So: what is the intended effect of :ASDF-DEPENDENCIES and :ASDF-OPTIONAL-DEPENDENCIES? At what point in the processing of the DEFSYSTEM are they processed? How do they interact with the processing of other DEFSYSTEM initargs? The answers to these questions should not be left to reading the code.

This depends on the level of implementation we achieve.

Ideally the situation could be such that the loading of dependencies could be delayed until the system is actually used. In other words, load the system definition using the base class, record components, dependencies, etc, Only when the system is actually loaded, compute the dependencies which are present and can be used and change the class using the extensions which are really available.

In practice it might not be that simple. There are situations right now in which it might have been useful to load the dependency _before_ even reading the system definition. These situations require further fixes:

- Right now people are using symbols in other packages. Solution: prevent this by specification; prefixed symbols should be strings and apply the previous rule

- Some systems use conditional reading #+/#- Solution: fix the use of :FEATURE in the DEFSYSTEM specification

- Some systems use #. plus functions they define Solution: use another readtable.

On Sun, Apr 18, 2010 at 5:24 PM, Robert Goldman rpgoldman@sift.info wrote:

On 4/18/10 Apr 18 -10:18 AM, Juan Jose Garcia-Ripoll wrote:

...

Ideally the situation could be such that the loading of dependencies could be delayed until the system is actually used. In other words, load the system definition using the base class, record components, dependencies, etc, Only when the system is actually loaded, compute the dependencies which are present and can be used and change the class using the extensions which are really available.

I was thinking one might do a two-phase process in which the :asdf-dependencies (and proposed :asdf-optional-dependencies) would be loaded first. Then the remaining defsystem initargs would be processed. Any reason not to do that?

This is actually how I implemented it, though the load step could be moved into the macro expansion itself, but I think it would be nice to also load system descriptions without having any further side effects, such as compiling and loading all of CFFI. If we impose that the dependencies and components have to be understood by ASDF we could create the dependency graph _without_ the :ASDF-DEPENDENCIES being even present (Thinking right now of an automated scan of *.asd files in some web based software repository and things like that).

...

• Right now people are using symbols in other packages.

This seems quite undesirable. You can't even /read/ such symbols reliably. Currently this works because the forms that load the asdf dependencies are lexically before the DEFSYSTEM form. But putting the asdf-dependencies there would make this impossible. We would need to find a solution much less cumbersome than the current

(funcall (intern <function name> <extension package>) ...)

or similar constructs which I find myself typing more often than I'd like in PERFORM methods...

Yes, this has to be sorted out. It is the real stopper towards purely declarative systems. From a simple inspection, the problem with packages seems right now constrained to - component names - :perform options in components (seems deactivated right now) - perform methods that are system-specific (EQL specialized) - some tuning of software variables / options / features

I have several ideas some of which amount to the following ingredients: + Reader macro magic + the expansion you mentioned + storing code as is + automated use of EVAL

The use of reader macros with some syntax ~absent-package::symbol could be used to keep a "to be constructed" form of symbol in an S-expression for the code that makes up the PERFORM methods and options (Alternatively we could use just strings) This could would then be evaluated or coerced into a lambda form only when needed -- and thanks to the dependency mechanism that would be when the packages do already exist.

Note that this is one of the reasons why I think that ASDF gets it wrong when writing rules which are specific to a system or to components that are only present in that system: in that case CLOS is an overkill and forces you to write methods _before_ we have installed any of the actual packages and functions the method depends on. Fixing this hack of storing system-dependent actions in the generic function PERFORM would be interesting mid-term goal.

Juanjo

On 4/19/10 Apr 19 -1:01 PM, Faré wrote:

I pushed Juanjo's patch for :ASDF-DEPENDENCIES as 1.704, except that I renamed the feature :SYSTEM-DEPENDENCIES. Maybe it should have been :SYSTEM-DEPENDS-ON. Ahem. You tell me what you think.

As for optional dependencies, I think things like ASDF-SYSTEM-CONNECTIONS are a walking nightmare, and horrible bugs waiting to happen. If you want to layer your system, just have foo-minimal or foo-core for the core functionality, and foo or foo-all for the whole system with all the dependencies. Trying to confuse the two under one name is but bugs waiting to happen.

I think that the test operation is the classic case where this method fails. The problem is that you want to specify /in the system to be tested/ the test methods, the test-dependencies, etc., but you don't want to force everyone who wants to use library X to install whatever library X's developer thought was the cool test framework.

optional dependencies are kinda nasty, but less nasty than:

(when (find-system :nst) (pushnew :nst-tester *features*))

(defsystem foo #+nst-tester #+nst-tester :in-order-to ((test-op (test-op foo-tester))) ...)

#+nst-tester (defsystem foo-tester :depends-on (nst-asdf foo) ....)

Is there a cleaner way to handle this?

Seems like a DOC-OP is almost certain to involve the same kind of nastiness....

Finally, considering the naming of things in the target package before it has been defined -- cl-launch usually stores forms like that as a string to be read and eval'ed after the system has been loaded. XCVB on the other hand has a call procedure that takes package + symbol designators as arguments to specify what to funcall after the package is ready.

Adding something like this to ASDF would be very nice. People shouldn't really have to do that (funcall (intern ...)) guff every time and, if we were to do this once, we could get it right once (and automagically handle the mlisp issues, while we are at it).

Best, r

I think defsystem-depends-on fits the current interface better indeed. If no one complains today, I'll commit that as 1.705. (Not that I like the current interface, but it's not mine to design.)

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] May your desire to be correct overcome your desire to have been correct (which you were not, anyway).

On 20 April 2010 04:10, Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com wrote:

On Mon, Apr 19, 2010 at 8:01 PM, Faré fahree@gmail.com wrote:

...

I pushed Juanjo's patch for :ASDF-DEPENDENCIES as 1.704, except that I renamed the feature :SYSTEM-DEPENDENCIES. Maybe it should have been :SYSTEM-DEPENDS-ON. Ahem. You tell me what you think.

There is obviously a naming problem.

:system-dependencies looks like the dependencies are equivalent to :depends-on :asdf-dependencies is ugly and does not give much information

I could suggest,

:meta-dependencies which is ugly, but may ring a bell (AMOP) :defsystem-depends-on which is more like a sentence (current spirit)

Juanjo

-- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

On Tue, Apr 20, 2010 at 6:17 PM, james anderson james.anderson@setf.dewrote:

that there is such difficulty to select a stable term indicates that the form of the expression is wrong.

Not really. It simply shows that some concepts are difficult to fit in one or two words -- as in :weakly-depends-on How weakly? Condition? Requirement? Works without, builds without?

the mechanism[1] which hard-wires parameter segregation in the defsystem macro is a tell-tale.

I do not like the current mechanism and I have made it explicit.

the implementation note[2] did not offer any reasons for the change,

This was a followup on various messages, as you have been able to find.

the asdf defsystem protocol fails these requirements in several ways:[...]

the instantiation protocol provides limited support for non-core class designators and requires that

• all packages must exist before the declaration is read,
• all classes must be defined before the declarations is interpreted.

Indeed this is a problem.

note that all of these issues apply to the model itself, not to the modeled application. in which case, the pertinent declaration forms should be distinct from those which apply to the system components.

Sorry, this paragraph is plain to abstract for me to understand. Indeed I find the discussion in general too abstract. It would be nice if you could formulate it in plain words that a reader of a "Users guide" can understand.

there are at least two ways to accomplish the syntactic machinations. one would be to follow the standard source patterns and standardize an independent expression

That would cause less pain because it could be macroexpanded, but it would fail to annotate the system form itself with a dependency, which I, against your way of thinking, is there -- just like a STANDARD-CLASS depends on the existence of such class to process its arguments, this is a meta-dependency: we need a system to parse the existing system. Would you then have the :metaclass argument outside DEFCLASS?

a second form would be to elaborate the system name by attaching the annotations to it.

Why the system name?

I insist. CLOS classes do have meta-annotations that give information on how to process the class definition itself, XML does as well, at least from a writer's point of view -- I do not know the programmer's side, which you know better having implemented a library for XML yourself.

there are several ways to support the late-binding for the system/

module/implementation classes. the existing asdf implementation already delays the process somewhat in the way it resolved classes by permitting the use of keywords. one can conceive of a protocol which relied on the same search process as applies to systems to be applied to the classes themselves during system construction. this would, however. mean that the current suggested practice of interpreting system declarations in a null package would be replaced with one in which the package was significant.

I do not agree. My view right now is the following:

1- system definitions continue to be read in the appropriate package. 2- at some point we may introduce tokens with late bindings to packages -- this is essential for rules that depend on functions that depend on packages that are not created. 3- we annotate the system definition with meta-dependencies. 4- We make _no_promise_, absolutely no promise about when the meta-dependencies will be use. In this sense this is exactly like the MOP class finalization protocol. 5- At some point we will decide to actually use the system for something else than querying dependencies or inspecting the semantic annotations (author, description, components, etc). 6- It is in that moment that meta dependencies will _typically_ be ensured, the system and components will be finalized to the appropriate classes and we will have fully working dependencies.

This is *not* what was implemented right now. What is implemented at the moment implicitly loads the dependencies before the system definition is parsed. However that still fits within the rule 4

Juanjo

On Tue, Apr 20, 2010 at 9:51 PM, james anderson james.anderson@setf.dewrote:

if the concepts are "difficult to fit in one or two words", then the expression form should disambiguate rather than obscure. just as (defstruct (person (:print-function print-person)) name age sex)

It is good to make more elaborate explanations. When you said "change the name structure" I understood something like (defsystem some-dependency-my-system ...) or something like that.

ia pattern such as (defsystem (a-system :class extended-system :require some-implementation) :components ((:file x))) is better than (defsystem a-system :class extended-system :require some-implementation :components ((:file x)))

I would have said that the latter is less different than current practice and thus it would have allowed for a simpler evolution of people's code. Your notation above is not at all disgusting, but it would change the way the :class argument is parsed right now.

if it might be possible to confirm that the shortlist, above, is a complete statement of the problem, that would help.

Your list seems mostly complete.

1 it should be possible to interpret a system definition without performing arbitrary computation 1a the interpretation includes constructing and/or interrogating the intra- and inter-system dependency graph 1b it includes identifying designated files 1c it should be possible to do this using ASDF's classes and parsers 1d it will be allowed to have additional "properties" or annotations for extensions to use 1e the system definition itself will be read in a "null" package, destroyed after reading the definition 1f* the system definition will not contain symbols in packages other than - CL - KEYWORD - The null package - ASDF - CL-USER (for setting configuration variables) 1g* If a symbol is needed from some other package it will be done with a special designator helped by a reader macro (~FUTURE-PACKAGE:SYMBOL-NAME or something like that). 1h* ASDF functions will recognize those designators and "intern" them in the appropriate packages only after dependencies are satisfied.

2 it should be possible to extend asdf operations on system, module, and file model components declaratively, in a manner such that the declaration still yield a valid definition in the absence of any extension 2a this includes to specify additional initargs in the specification 2b this includes to specify classes which are not defined in the asdf core.

3 Guarantees about extensions 3a The extensions will be available when the first operation is PERFORMed on the system or on any of its components. By then the system and component instances will have been processed and finished by the extensions. 3b In particular ASDF does not promise that the extensions will be available when reading 3c Those extensions will be loaded using ASDF's mechanisms and may be reloaded if their own dependencies change. 3d Reloading extensions may result in an update of the system classes and components and extensions should be coded to allow instance reinitialization. 3e** Extensions should not change the list of dependencies of the components as previously computed by ASDF. However, they are allowed to insert harmless dependencies in the edges or as hanging leaves, such as intermediate actions to be performed, as far as they do not change the dependencies between components and systems.

(*) Some of these things would require further work, the rest is mostly there. (**) This could be formulated in a more rigorous way.

[... listing the dependencies outside DEFSYSTEM ...] That would cause less pain because it could be macroexpanded, but it would fail to annotate the system form itself with a dependency,

is that one of the requirements? one of the earlier discussions suggested that the load protocol be implemented as a restricted read-eval loop. which would itself be in a position to collect any annotations.

If you consider the system definition as the whole file, then I agree with you that it really does not matter where things are placed. I just find it more appealing to have just one thing to read: the DEFSYSTEM form, where all annotations and dependencies are placed together -- just like you find it more natural to have the meta-information in the name.

it depends on whether the requirement is that it behave like use/in-package or like :metaclass. given the present formulation, what is the consequence of a dependency which appears in more than one system definition. is it reloaded or not? what happens if the features have changed since it was first loaded? is the behavior by intent, or a side-effect of the implementation?

Even metaclasses can change themselves. What I do not want to emphasize is the view that :asdf-dependencies or :defsystem-depends-on or whatever formula we arrive at is viewed as an imperative USE-PACKAGE that is used _before_ the defsystem form itself and before it is read. It is just part of it, listing where system classes, component classes and perhaps additional parsing routines live, and informing us that they will be used to create the system itself, somehow similar to how a metaclass handles creation, instantiation, access to slots, etc.

1- system definitions continue to be read in the appropriate package.

as of last reading, they were loaded by default into a temporary package which was then deleted. is this the appropriate package? if so, why?

I find it appropriate because it is a temporary package and we can detect symbols that are pulled from other packages, issuing a warning if needed.

2- at some point we may introduce tokens with late bindings to packages -- this is essential for rules that depend on functions that depend on packages that are not created.

why tokens? what function do they support and why are tokens the best way to accomplish that? why is it better to specify the future home package that it would be to search for it? why is that better than it would be to always use the same package? why is it better than to use a package named after the system?

The defsystem form has to have some way of referring to future packages because we are going to need it. We have a file "generated-lisp.lsp" which is listed in the defsystem

(defsystem :example :components ((:file "dsl-parser") (:file "generated" :create (~dsl-parser:transform "generated-source.dsl" "generated.lisp"))))

Please do not take this literally: I do not mean this to be the right syntax. However you appreciate the need for a rule which is not general, not reusable, just to create this file. This rule needs a function that does not exist when we create the system and which has a name that lives in a package that also does not exist. If we allow this kind of "tokens" (is this the right word?) and teach ASDF to generate the right substitution when the dependencies are satisfied (in this case after dsl-parser is loaded and when generated is created), then we can be happy.

This is something we will face with this example or with other ones that have dependencies in non-existent packages. For instance components such as the ones defined by CFFI-GROVEL. This time taken from a real example:

(asdf:defsystem :fsbv :description "Foreign Structures By Value." :maintainer "Liam Healy lhealy@common-lisp.net" :licence "See readme.html"

:depends-on (:cffi :cffi-grovel :trivial-features) ;;:pathname (merge-pathnames "syscalls/" *load-truename*) :serial t :components ((:file "init") (cffi-grovel:grovel-file "libffi" :pathname #+unix "libffi-unix")

We could list CFFI-GROVEL as a meta dependency, but even then we would not be able to read this form without loading the extension. If we instead allow for some kind of "future package" designator, then we could.

3- we annotate the system definition with meta-dependencies.

what are meta-dependencies?

• packages?
• classes in packages?
• systems?
• features?
• ?

So far I have only talked about two: classes (:class) and libraries that implement extensions (:asdf-dependency)

4- We make _no_promise_, absolutely no promise about when the meta-dependencies will be use. In this sense this is exactly like the MOP class finalization protocol.

before perform? before operate-on-system? before shared-initialize starts? completes? before initialize-instance starts, completes?

This is debatable and it will depend how much liberty we want to give to extension writers. I would argue that just before PERFORM. We should be able to traverse the dependencies, create the appropriate graphs, etc, using just ASDF's classes, without having the extensions around.

The problem I see is that people may find this too limiting. For instance I will take a real life example: a user that creates his own CSS-FILE class because STATIC-FILE does not provide extensions (defclass css-file (doc-file) ()) (defmethod source-file-type ((c css-file) (s module)) "css") used as follows (:module "doc" :components ((:html-file "ironclad") ;; XXX ASDF bogosity (:css-file "style")))))

If we enforce the policy that the extension is not enforced by TRAVERSE, this class will not be loaded and we will fail to recognize that the system depends on "style.css" and not on "style".

Now one may argue precisely that this particular example is artificial, because if he used the keyword :type "css" he could have used the :doc-file component instead. But I believe that indeed this shows that if we provide a rich, expressive and concise defsystem grammar by default, then extensions may leave the handling of dependencies to ASDF and focus on other things: how actual operations are performed.

I am sorry I was too verbose, as usual, and you will by now totally bored, but I hope you will not find this writing void of logic, as you usually do :-)

Juanjo