This is an idea that has been long floating in the back of my mind, and was brought back to life by these comments http://groups.google.com/group/comp.lang.lisp/msg/f99a69797eda1caf
The problem is that many people use *.asd files to do things like building up packages, creating operations, defining methods, etc. That makes it impossible to ship those libraries because the side effects are not recorded in the system itself. So if ECL traverses the system to build a standalone program it will know which sources are required, but those sources may for instance rely on a package that was created in the *.asd definition.
My suggestion is a gradual move towards enforcing that *.asd files become mere descriptions of the system and that side effects needed for building and loading are listed in the system itself.
- System definitions should be read and recorded by ASDF in pure form, that is as S expressions, with a hash that maps system names to s-expressions. - Instead of using LOAD we use a LOOP and READ. If a form which is not a DEFSYSTEM is found, we have two options: if we enforce strict mode, an error is signaled, otherwise the form is evaluated after emitting a warning. - Warnings could also be emitted when a DEFSYSTEM clause does not match the name of the file, or when more than one such clauses is found. - The system definition should be scanned for a class name and for a keyword option, :asdf-support If the :asdf-support option is found, it should name a file with side effects that are needed for the ASDF definition to be used. The file should be loaded using LOAD. - The original S expression is now parsed and the system definition is created.
The warning could be really self-explanatory, stating the following: - Dependencies that are needed for processing the system definition must be listed in :asdf-support - Dependencies that are needed for load/compile-op should be listed in :depends-on
I can think on even more elegant ways to do this. For instance having proxy classes, such as proxy-system and proxy-component which are used to store the S-expression for the system and using change-class on these objects to transform them into valid systems and components on demand, that is only when the systems are needed.
It should also be possible to write an utility that grovels existing ASDF systems looking for potential problems and suggesting changes.
What the solution above does not address is the following sentence from Daniel's comment:
As to the "no". Simply doing asdf:find-component may result in side effects, both in the package of interest and in random other packages before it. These side effects include dumping fasls. There are several asd files that force things to load early in order to extend ASDF by defining custom classes."
In other words, the side effects in :asdf-support can be really harmful, or costly. However this proposal could help us identify what libraries do have side effects and also help enforce further restrictions in the side effects they may have.
I mean, there are many many other ways to do this, but I think you get the idea.
Juanjo
I can see Juanjo's point, but I would like to suggest that we defer this until ASDF 3; it's simply too much to bite off this time around, IMO.
Suggestion: if the group thinks this is a good idea, let's
1. Add a "future plans" section to the manual
2. Insert a write-up about this proposal in there.
Best, Robert
On Thu, Mar 18, 2010 at 10:04 PM, Robert Goldman rpgoldman@sift.infowrote:
I can see Juanjo's point, but I would like to suggest that we defer this until ASDF 3; it's simply too much to bite off this time around, IMO.
I agree that it is hard to get this right. However, would it be ok if I code a crawler that performs the anaylisis of ASDF systems? This would be a nice excuse to add a collect-op and ensure it works.
Suggestion: if the group thinks this is a good idea, let's
- Add a "future plans" section to the manual
- Insert a write-up about this proposal in there.
Would a wiki make sense? I built one for ecl in wikispaces.com and it is quite easy to handle and change by anyone -- not only by people with access to the manual sources.
Best,
Juanjo
On 3/18/10 Mar 18 -4:31 PM, Juan Jose Garcia-Ripoll wrote:
On Thu, Mar 18, 2010 at 10:04 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:
I can see Juanjo's point, but I would like to suggest that we defer this until ASDF 3; it's simply too much to bite off this time around, IMO.I agree that it is hard to get this right. However, would it be ok if I code a crawler that performs the anaylisis of ASDF systems? This would be a nice excuse to add a collect-op and ensure it works.
Suggestion: if the group thinks this is a good idea, let's 1. Add a "future plans" section to the manual 2. Insert a write-up about this proposal in there.Would a wiki make sense? I built one for ecl in wikispaces.com http://wikispaces.com and it is quite easy to handle and change by anyone -- not only by people with access to the manual sources.
I'm reluctant to endorse this suggestion, because it means that we will spawn YA information source to track. We'll have:
1. launchpad 2. mailing list 3. manual 4. wiki
Question: would it be acceptable to you to use the launchpad as a quasi-wiki for this? We could ticket this as an "enhancement," if that would meet with your approval.
I see the advantages of a wiki over launchpad, but the diffusion seems bad...
Best, r
On Thu, Mar 18, 2010 at 10:48 PM, Robert Goldman rpgoldman@sift.infowrote:
I'm reluctant to endorse this suggestion, because it means that we will spawn YA information source to track. We'll have:
- launchpad
I ignored ASDF was using this. I have submitted a wishlist ticket but I was not able to find a way to tag it as a wish.
- mailing list
The mailing list can not be considered as a source of information, things written here get lost in a humongous archive without any order and sometimes even without very informative subjects.
- manual
This is static information, without any
- wiki
Question: would it be acceptable to you to use the launchpad as a quasi-wiki for this? We could ticket this as an "enhancement," if that would meet with your approval.
Juanjo
On 3/18/10 Mar 18 -5:27 PM, Juan Jose Garcia-Ripoll wrote:
On Thu, Mar 18, 2010 at 10:48 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:
I'm reluctant to endorse this suggestion, because it means that we will spawn YA information source to track. We'll have: 1. launchpadI ignored ASDF was using this. I have submitted a wishlist ticket but I was not able to find a way to tag it as a wish.
Seems like you can do this under "importance," so I did that.
Regarding wish lists for ASDF 3, you know where to find the XCVB mailing-list and XCVB's TODO file when I need them.
And while I agree that manually converting ASDF systems that can't be automatically converted is not always an option, XCVB lets you use ASDF systems as is, and could be modified to mix-and-match XCVB builds and ASDF systems even more easily than now.
In other words, I think that supporting ASDF is important, but ultimately an evolutionary dead-end.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Government's view of the economy could be summed up in a few short phrases : If it moves, tax it. If it keeps moving, regulate it. And if it stops moving, subsidize it. — Ronald Reagan (1986)
On Thu, Mar 18, 2010 at 11:28 PM, Faré fahree@gmail.com wrote:
In other words, I think that supporting ASDF is important, but ultimately an evolutionary dead-end.
That is hard to read from the project leaders. I feel rather neutral about this and please do not feel offended by the following sentence, but if this is going to be the official position of the project then it should be better advertised, including perhaps the manual, usual c.l.l forums and maybe the ASDF prompt itself.
Juanjo
On 18 March 2010 18:37, Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com wrote:
On Thu, Mar 18, 2010 at 11:28 PM, Faré fahree@gmail.com wrote:
In other words, I think that supporting ASDF is important, but ultimately an evolutionary dead-end.
That is hard to read from the project leaders. I feel rather neutral about this and please do not feel offended by the following sentence, but if this is going to be the official position of the project then it should be better advertised, including perhaps the manual, usual c.l.l forums and maybe the ASDF prompt itself.
No, this is not an official position as an "ASDF project leader". Moreover, I do intend to resign from active ASDF development after ASDF 2 is released, and focus my CL build hacking activity wholly on XCVB.
Inasmuch as there is an "official position" of "ASDF project leaders" as such, it is that ASDF will be the best ASDF it can be, and that XCVB is not on the radar. (Unofficially, we tailored our source-registry API so it may be directly reused by XCVB.)
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] If it doesn't work right, we can always try something else. — John McCarthy
On 18 March 2010 16:41, Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com wrote:
This is an idea that has been long floating in the back of my mind, and was brought back to life by these comments http://groups.google.com/group/comp.lang.lisp/msg/f99a69797eda1caf
The problem is that many people use *.asd files to do things like building up packages, creating operations, defining methods, etc. That makes it impossible to ship those libraries because the side effects are not recorded in the system itself. So if ECL traverses the system to build a standalone program it will know which sources are required, but those sources may for instance rely on a package that was created in the *.asd definition.
My suggestion is a gradual move towards enforcing that *.asd files become mere descriptions of the system and that side effects needed for building and loading are listed in the system itself.
What about instead investing in XCVB?
Such enforcement will necessarily introduce backward incompatibility and pain, which I think goes contrary to the goals of ASDF.
XCVB, on the other hand, is designed to pretty much do what you ask for: have system descriptions be pure information manipulated by an external process without uncontrolled side-effects.
As to systems that currently use weird ASDF extensions, you could either make XCVB's ASDF converter better, or just convert these systems by hand.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The World owes you nothing. You owe everything to yourself.
On Thu, Mar 18, 2010 at 10:47 PM, Faré fahree@gmail.com wrote:
What about instead investing in XCVB?
As much as I would like to have something simplify my life, it is not my choice to use one system or another. I was suggesting something that might help rationalize the current library situation.
Such enforcement will necessarily introduce backward incompatibility and pain, which I think goes contrary to the goals of ASDF.
I am not talking about something that HAS to be enforced, but that it can be optional. It is by no means a good coding practice to put things in the *.asd file that do not belong in it. Promoting this message from the ASDF development forum is not wrong by itself and need not cause any pain at all -- a warning message somewhere in the build, explaining the situation of the libraries in the system does not cause such a panic or disrupture, does it?
I think it should be spoken clearly here about what is expected also from the ASDF maintainers. Either you envision it as a hack until something better comes along, or you consider the possibility of gradually evolving towards something better.
As to systems that currently use weird ASDF extensions, you could either make XCVB's ASDF converter better, or just convert these systems by hand.
Conversion by hand is not an option. No time and no will on my side to track all libraries written out there. I also do not believe that it would help without support on the other side of the line: this should be done by the developers themselves once they discover that a build system provides some advantage.
Juanjo
On 3/18/10 Mar 18 -5:11 PM, Juan Jose Garcia-Ripoll wrote:
On Thu, Mar 18, 2010 at 10:47 PM, Faré <fahree@gmail.com mailto:fahree@gmail.com> wrote:
What about instead investing in XCVB?As much as I would like to have something simplify my life, it is not my choice to use one system or another. I was suggesting something that might help rationalize the current library situation.
Such enforcement will necessarily introduce backward incompatibility and pain, which I think goes contrary to the goals of ASDF.I am not talking about something that HAS to be enforced, but that it can be optional. It is by no means a good coding practice to put things in the *.asd file that do not belong in it. Promoting this message from the ASDF development forum is not wrong by itself and need not cause any pain at all -- a warning message somewhere in the build, explaining the situation of the libraries in the system does not cause such a panic or disrupture, does it?
Right. But do we have a clear understanding of what should and shouldn't go in there? E.g.:
1. currently if you need an ASDF extension in order to make a defsystem understandable (e.g., we have an extension that provides a new SYSTEM subclass to fit with our testing library), then you must put
(asdf:oos 'asdf:load-op <my extension>)
in the .ASD file, and this is undesirable, I agree, but there's no workaround.
2. New class and method definitions. We don't have a good way to put them anywhere /but/ the .asd file for now.
I see the point about good coding practice, but I feel weird about telling people to use good coding practice at the same time telling them they have to use bad (non-declarative) coding practice, because there's no alternative!
Can you say more about what you'd like to do specifically? I don't want to discourage you from providing support for the sad lot of ASDF system definers ;-)!
On Thu, Mar 18, 2010 at 11:18 PM, Robert Goldman rpgoldman@sift.infowrote:
Right. But do we have a clear understanding of what should and shouldn't go in there? E.g.:
- currently if you need an ASDF extension in order to make a defsystem
understandable [...] 2. New class and method definitions. We don't have a good way to put them anywhere /but/ the .asd file for now.
I see the point about good coding practice, but I feel weird about telling people to use good coding practice at the same time telling them they have to use bad (non-declarative) coding practice, because there's no alternative!
Can you say more about what you'd like to do specifically? I don't want to discourage you from providing support for the sad lot of ASDF system definers ;-)!
Please understand that I did not intend to prevent people from writing their own system or operation classes. That would go against my own practice :-) If you read my email, it is for this reason that I explicitely added an :asdf-support file option, where ASDF extensions should be coded.
The problem, as I said, it is not extensions per se because they are needed to build the system. The problem is when people beging coding additional stuff -- I mentioned packages, but I have found classes, functions and other things that are not related to ASDF but are actually used by the code that the ASDF system is describing.
Juanjo
Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com writes:
This is an idea that has been long floating in the back of my mind, and was brought back to life by these comments http://groups.google.com/group/comp.lang.lisp/msg/f99a69797eda1caf
The problem is that many people use *.asd files to do things like building up packages, creating operations, defining methods, etc. That makes it impossible to ship those libraries because the side effects are not recorded in the system itself. So if ECL traverses the system to build a standalone program it will know which sources are required, but those sources may for instance rely on a package that was created in the *.asd definition.
I suggest that the Manual contains a few notes on Style, perhaps as an FAQ. The notes should also explain why pureness in .asd is worthwhile. I'll submit a bug entry for this.
Also FAQ 12.5.1 "How can I cater for unit-testing in my system?", does not actually fully show how to add unit-testing in pure style; it shows the IN-ORDER for the FOO system, but does not show how to actually invoke the RUN-TESTS (or whatever) in the FOO-TEST system. I'll submit a bug entry for this, too.
There, however, seems to be an inherent dependency-vs-pureness problem with user extensions as illustrated by cffi-grovel; from [1]:
;;; CFFI-Grovel is needed for processing grovel-file components (cl:eval-when (:load-toplevel :execute) (asdf:operate 'asdf:load-op 'cffi-grovel))
(asdf:defsystem example-software :depends-on (cffi) :serial t :components ((:file "package") (cffi-grovel:grovel-file "example-grovelling") (:file "example")))
I know Stelian cursed about this in-persona, wishing for a reader which has a notion of unresolved-symbols.
-T.
[1] http://common-lisp.net/project/cffi/manual/html_node/Groveller-ASDF-Integrat...
On Sun, Mar 21, 2010 at 12:41 PM, Tobias C. Rittweiler tcr@freebits.dewrote:
There, however, seems to be an inherent dependency-vs-pureness problem with user extensions as illustrated by cffi-grovel; from [1]:
;;; CFFI-Grovel is needed for processing grovel-file components (cl:eval-when (:load-toplevel :execute) (asdf:operate 'asdf:load-op 'cffi-grovel))
(asdf:defsystem example-software :depends-on (cffi) :serial t :components ((:file "package") (cffi-grovel:grovel-file "example-grovelling") (:file "example")))
I know Stelian cursed about this in-persona, wishing for a reader which has a notion of unresolved-symbols.
This can be cured with my suggestions plus some extensions I note below
- Add a field :asdf-support to list dependencies for the system itself. - Add a feature by which component types are registered with ASDF so that they can be named using keywords - Impose that parsing of the defsystem list is done _after_ loading the support components themselves.
Juanjo
On 3/21/10 Mar 21 -11:55 AM, Juan Jose Garcia-Ripoll wrote:
On Sun, Mar 21, 2010 at 12:41 PM, Tobias C. Rittweiler <tcr@freebits.de mailto:tcr@freebits.de> wrote:
There, however, seems to be an inherent dependency-vs-pureness problem with user extensions as illustrated by cffi-grovel; from [1]: ;;; CFFI-Grovel is needed for processing grovel-file components (cl:eval-when (:load-toplevel :execute) (asdf:operate 'asdf:load-op 'cffi-grovel)) (asdf:defsystem example-software :depends-on (cffi) :serial t :components ((:file "package") (cffi-grovel:grovel-file "example-grovelling") (:file "example"))) I know Stelian cursed about this in-persona, wishing for a reader which has a notion of unresolved-symbols.This can be cured with my suggestions plus some extensions I note below
- Add a field :asdf-support to list dependencies for the system itself.
- Add a feature by which component types are registered with ASDF so
that they can be named using keywords
Are you sure you can't do this already? I'm looking at class-for-type, and it tries to look up the symbol-name of the component class name in the keyword package.
Please check (I've got a big project today; sorry, I can't spare the time to test this).
Cheers, r
On Sun, 21 Mar 2010, Robert Goldman wrote:
On 3/21/10 Mar 21 -11:55 AM, Juan Jose Garcia-Ripoll wrote:
- Add a field :asdf-support to list dependencies for the system itself.
- Add a feature by which component types are registered with ASDF so
that they can be named using keywords
Are you sure you can't do this already? I'm looking at class-for-type, and it tries to look up the symbol-name of the component class name in the keyword package.
If that's the case, then its a well-kept secret that needs advertising. Either way, it would still be good to add the ":asdf-support" list telling asdf which packages to load for each extension.
Maybe a clause at the top of the file like (asdf:extensions (defsystem-connection asdf-system-connections) (other-extension other-package))
I don't think this needs to be put in the system.
Later, Daniel
On 3/21/10 Mar 21 -9:16 PM, Daniel Herring wrote:
On Sun, 21 Mar 2010, Robert Goldman wrote:
On 3/21/10 Mar 21 -11:55 AM, Juan Jose Garcia-Ripoll wrote:
- Add a field :asdf-support to list dependencies for the system itself.
- Add a feature by which component types are registered with ASDF so
that they can be named using keywords
Are you sure you can't do this already? I'm looking at class-for-type, and it tries to look up the symbol-name of the component class name in the keyword package.
If that's the case, then its a well-kept secret that needs advertising. Either way, it would still be good to add the ":asdf-support" list telling asdf which packages to load for each extension.
Done. I forget how I knew that you could do this.... Anyway, I have just documented this in "the defsystem grammar." It would probably be a good idea to have an example of this in one of the defsystem examples, if anyone can supply one.
Alternatively, if no one feels like supplying one, I'm not in good internet reach right now; feel free to request such an example by posting a launchpad ticket and I'll try to get one done.
Good night, all!
R
good morning;
On 2010-03-18, at 21:41 , Juan Jose Garcia-Ripoll wrote:
This is an idea that has been long floating in the back of my mind, and was brought back to life by these comments http://groups.google.com/group/comp.lang.lisp/msg/f99a69797eda1caf
The problem is that many people use *.asd files to do things like building up packages, creating operations, defining methods, etc. That makes it impossible to ship those libraries because the side effects are not recorded in the system itself. So if ECL traverses the system to build a standalone program it will know which sources are required, but those sources may for instance rely on a package that was created in the *.asd definition.
My suggestion is a gradual move towards enforcing that *.asd files become mere descriptions of the system and that side effects needed for building and loading are listed in the system itself.
JJ.Garcia-Ripoll posed a question central to asdf: should it enforce 'pure' system definitions? At first, the suggestion seemed unrealistic. Upon reflection, it remains at odds with the prevailing reality, but it reveals that this reality is not the one which asdf should choose. A bit of reflection and a small experiment compel the realization, that asdf is not unix and should not persist as if it were. It should concentrate on describing lisp programs.
An .asd form is an abstract description of a lisp program, of the program's components, their construction ordering constraints, and their locations. The asdf facility should be able to use any valid .asd forms to infer that information about a system, use it to construct the system, and report the information to an application. Nothing else matters. If a form cannot be used for that purpose, it is invalid. asdf implementation has two essential aspects. First, to interpret the definition forms to construct an annotated graph which models the ordering, location, and operation information. Second, to use that graph to construct the program. Under present circumstances, for between ten and twenty percent of open-source lisp projects, asdf does not satisfy even the first requirement. This is perhaps not disgraceful, but well more than lamentable. asdf in its present form and in the present .asd ecosystem is by no means capable to perform the task which distinguishes its purpose. We need to fix that. Fix it before one progresses to version 3, or version 2, or even version 1.7.
One may object, that the suggestion is unrealistic. That it is impossible and would sacrifice backwards compatibility. I expect the first assertion to turn out to be false. The second demonstrates the central problem, that lisp developers have had to learn to live with asdf inadequacies. Somehow. In the process, .asd forms have evolved to compensate for essential deficiencies. In order to work around deficiencies, developers use .asd files to extend the build facility itself simultaneous with defining the program to be built. To the extent, that the actual system definition is incidental. In some cases [0] to the point that the thought occurs, "maybe it would be easier to map some load equivalent over a file list."
Several asdf deficiencies require these practices. All should be corrected and the practices deprecated. To the effect that they are not supported in future asdf versions. One might even consider to back-port the corrections, but that might prove unpopular.
The primary problem is that there is no way to declare extensions. As bemoaned by T.Rittweiler with deference to S.Ionescu, and demonstrated by projects which use extensions provided by cffi- grovel, asdf-system-connections, as well as projects like cl-perec and steffil, which define extension classes in-line, there is neither a form which declares the system definition's dependency on an extension, nor one which incorporates an extension. In the case of the last two projects, they choose to resort to loading the .asd files into project-specific packages in order to to distinguishing homonyms. Yes, it is possible. The effect of such license is that the asdf facility cannot interpret a significant percentage of the system descriptions in the wild without running arbitrary computations.[1] Yes, it is possible, but there is no reason to tolerate it. In particular, because this is not difficult to fix. use-package would suffice.
Contributing to this problem is that the system and component forms are not symmetrical. The intended syntax is[2], but the implementation is not:
? (defclass specialized-system (asdf:system) ()) #<STANDARD-CLASS SPECIALIZED-SYSTEM> ? (defclass specialized-source-file (asdf:cl-source-file) ()) #<STANDARD-CLASS SPECIALIZED-SOURCE-FILE> ? (eval '(asdf:defsystem :a-system :class specialized- system :components ((:file "a-file")))) #<SPECIALIZED-SYSTEM "a-system" #xEF6724E> ? (eval '(asdf:defsystem :a-system :class specialized- system :components ((:file "a-file" :class specialized-source-file))))
Error: :CLASS is an invalid initarg to REINITIALIZE-INSTANCE for
#<STANDARD-CLASS ASDF:CL-SOURCE-FILE>.
Valid initargs: #(:CONTINGENT-ON :LONG-
DESCRIPTION :DESCRIPTION :NAME :VERSION :IN-ORDER-TO :DO- FIRST :PARENT :PATHNAME :PROPERTIES).
While executing: CCL::CHECK-INITARGS Type Command-. to abort.
See the Restarts… menu item for further choices. 1 >
This should be corrected, in order that - even in the absence of an intended extension, asdf can interpret the standard information in a system definition.
A related issue, is that, without the extension, a specialization's initialization arguments would not be permitted. This should be corrected as well.
An a-bit-more-than thought experiment issued from the following use case. Would it be possible to perform some kind of introspective analysis of the lisp open-source ecosytem without opening a server to arbitrary computation? Something in the same realm as clbuild, but without the script maintenance. The short answer: to a large percent yes, but there are some projects which reside well beyond the reach of such a service.
It was a simple experiment. Define a restricted package and interpret .asd files with a restricted read-eval loop.
[ continued ... ]
On 3/23/10 Mar 23 -8:35 AM, james anderson wrote:
good morning;
On 2010-03-18, at 21:41 , Juan Jose Garcia-Ripoll wrote:
Contributing to this problem is that the system and component forms are not symmetrical. The intended syntax is[2], but the implementation is not:
? (defclass specialized-system (asdf:system) ()) #<STANDARD-CLASS SPECIALIZED-SYSTEM> ? (defclass specialized-source-file (asdf:cl-source-file) ()) #<STANDARD-CLASS SPECIALIZED-SOURCE-FILE> ? (eval '(asdf:defsystem :a-system :class specialized-system :components ((:file "a-file")))) #<SPECIALIZED-SYSTEM "a-system" #xEF6724E> ? (eval '(asdf:defsystem :a-system :class specialized-system :components ((:file "a-file" :class specialized-source-file))))
Error: :CLASS is an invalid initarg to REINITIALIZE-INSTANCE for
#<STANDARD-CLASS ASDF:CL-SOURCE-FILE>.
Valid initargs: #(:CONTINGENT-ON :LONG-DESCRIPTION :DESCRIPTION:NAME :VERSION :IN-ORDER-TO :DO-FIRST :PARENT :PATHNAME :PROPERTIES).
While executing: CCL::CHECK-INITARGS Type Command-. to abort.
See the Restarts… menu item for further choices. 1 >
This should be corrected, in order that - even in the absence of an intended extension, asdf can interpret the standard information in a system definition.
Please launchpad this bug, with a list of the set of initargs that need to be supported. Maybe this can be fixed.
I am willing to believe that the initargs to system should be supported on components to the maximum extent possible. But I don't see any reason, a priori, that components and systems should be treated symmetrically. Why should we believe that, of necessity, every form of source file should be modeled by a data structure that shares every attribute of a system? To take an absurd case, we don't put file extensions on systems....
Actually, I'm pretty shocked by the extent that source files /already/ share the attributes of systems. For example, source files, as components, have :version attributes. But surely if one wanted to have versions on source-files, there should be some inheritance relationship between the :version one specifies on a parent system and the version specified for its components. But we make no attempt to do any such propagation. Nor, since the :version is a property that lives in the system definition, and since the :version property of a component is not readily checkable across system boundaries, does associating versions with components make any particular sense.
The :class case, I'll grant you, /does/ make sense, but I'm not confident we should extrapolate too far from that.
best, r
On 2010-03-24, at 00:08 , Robert Goldman wrote:
On 3/23/10 Mar 23 -8:35 AM, james anderson wrote:
good morning;
On 2010-03-18, at 21:41 , Juan Jose Garcia-Ripoll wrote:
Contributing to this problem is that the system and component forms are not symmetrical. The intended syntax is[2], but the implementation is not:
? (defclass specialized-system (asdf:system) ()) #<STANDARD-CLASS SPECIALIZED-SYSTEM> ? (defclass specialized-source-file (asdf:cl-source-file) ()) #<STANDARD-CLASS SPECIALIZED-SOURCE-FILE> ? (eval '(asdf:defsystem :a-system :class specialized- system :components ((:file "a-file")))) #<SPECIALIZED-SYSTEM "a-system" #xEF6724E> ? (eval '(asdf:defsystem :a-system :class specialized- system :components ((:file "a-file" :class specialized-source-file))))
Error: :CLASS is an invalid initarg to REINITIALIZE-INSTANCE for
#<STANDARD-CLASS ASDF:CL-SOURCE-FILE>.
Valid initargs: #(:CONTINGENT-ON :LONG-DESCRIPTION :DESCRIPTION
:NAME :VERSION :IN-ORDER-TO :DO-FIRST :PARENT :PATHNAME :PROPERTIES).
While executing: CCL::CHECK-INITARGS Type Command-. to abort.
See the Restarts… menu item for further choices. 1 >
This should be corrected, in order that - even in the absence of an intended extension, asdf can interpret the standard information in a system definition.
Please launchpad this bug, with a list of the set of initargs that need to be supported. Maybe this can be fixed.
I am willing to believe that the initargs to system should be supported on components to the maximum extent possible. But I don't see any reason, a priori, that components and systems should be treated symmetrically. Why should we believe that, of necessity, every form of source file should be modeled by a data structure that shares every attribute of a system? To take an absurd case, we don't put file extensions on systems....
please reconsider whether, that one does not now really means that the particular feature should not be treated symmetrically.
i can't answer the question, as i've not considered the implications, but, given how obviously ill-considered the :class treatment is, i would first presume symmetry and exclude upon deliberation. ? what happens of one loads a system definition from a file which is not marked type "asd"?
Actually, I'm pretty shocked by the extent that source files /already/ share the attributes of systems.
one should more often boggle ones mind.
For example, source files, as components, have :version attributes. But surely if one wanted to have versions on source-files, there should be some inheritance relationship between the :version one specifies on a parent system and the version specified for its components.
given how little i comprehend the version's practical effects, i would not neither deny nor confirm that.
But we make no attempt to do any such propagation.
i would, however, not admit that such a practice, in itself, indicates what should or should not be done.
Nor, since the :version is a property that lives in the system definition, and since the :version property of a component is not readily checkable across system boundaries, does associating versions with components make any particular sense.
is there some inherent reason, by which version-ness does not propagate upwards? for example from the files which apply given implementation conditionalization to the system from which they are built? as noted, while unaware of how people actually use it, i would not presume asymmetry.
The :class case, I'll grant you, /does/ make sense, but I'm not confident we should extrapolate too far from that.
if one would go through each initialization argument and consider the implications of each, i would be more inclined to believe.
On 3/23/10 Mar 23 -8:01 PM, james anderson wrote:
On 2010-03-24, at 00:08 , Robert Goldman wrote:
On 3/23/10 Mar 23 -8:35 AM, james anderson wrote:
good morning;
On 2010-03-18, at 21:41 , Juan Jose Garcia-Ripoll wrote:
....
The :class case, I'll grant you, /does/ make sense, but I'm not confident we should extrapolate too far from that.
if one would go through each initialization argument and consider the implications of each, i would be more inclined to believe.
Right. Let me be a bit more specific. Any time I have to devote to ASDF between now and a hopefully soon-to-come version 2, is going to be sucked up in my launchpad backlog (especially the documentation items), which sadly seems to be monotonically increasing in size.
So if you or someone else on the list were to do that pass over the initialization arguments and, say, put them into a table with rows for arguments and columns for, say, source-file, (internal) module, and system, it would be hugely helpful.
I understand, of course, that you have already contributed hugely to the project, and you have your own life to lead, so it might well not be feasible for you.
Best, Robert
[ ... continued ]
It was a simple experiment. Define a restricted package and interpret .asd files with a restricted read-eval loop.
(defpackage :asdf-prototype (:use ) (:documentation "A constrained package to use for loading asdf system prototypes.") (:export :intern) (:import-from :asdf :system :file :module :cl-source-file :defsystem :compile-op :load-op :test-op) (:import-from :common-lisp :*load-truename* :symbol-name :with-open-file :merge-pathnames :make-pathname :read :describe :string))
(defun asdf-prototype::funcall (&rest args) "Serves as a place-jolder in order to define systems, but should never be called." (error "attempted funcall: ~s." args))
(defun asdf-prototype::intern (string &optional package) (declare (ignore package)) (intern string :asdf-prototype))
(defun load-system-prototype (pathname) (with-open-file (input pathname :direction :input) (with-standard-io-syntax (let ((*package* (or (find-package :asdf-prototype) (error "No prototype package."))) ;;(*read-eval* nil) (*print-readably* nil) (form nil) (definition nil) (*load-truename* (truename pathname)) (system nil) (error nil) (repository-url-properties nil)) (loop (setf form (read input nil input)) (when (eq form input) (return)) (when (consp form) (when (eq (first form) 'asdf:defsystem) (setf definition form) (multiple-value-setq (system error) (ignore-errors (eval definition))) (when error (warn "Loading system definition causes an error: ~s,~%~a" pathname error)) (when system (setf (find-system-prototype (asdf:component- name system)) system) (setf (definition-form system) definition) (setf (definition-pathname system) pathname) (when (setf repository-url-properties (ignore-errors (load-repository-url (make- pathname :name "repository" :t ype "url" :d efaults (definition-metadata-directory system))))) (destructuring-bind (&key name url type &allow- other-keys) repository-url-properties (when (and (string-equal name (asdf:component-name system)) url type) (setf (system-repository-url system) url (system-repository-type system) type))))))))
(values system error)))))
Just in order to see how much it would cover, *read-eval* is left as t. Better would be to disable it. It included neutered intern and funcall operators to facilitated loading. They indicate the presence of idioms which should be eliminated.
Mostly it was uneventful. Which is nice. Of the 91 projects present in the clbuild wnpp-projects file, 84 system definitions loaded. Because some systems include numerous nominally indistinguishable .asd files, the ratio is not conclusive, but it is still heartening. I observed a bit more than a dozen failures. The most prevalent failure was reference to a specialized component class. None of which would be necessary if it were possible to declare extensions. Of the specialized component implementations, local-cl-source-file and system-with-readtable indicate capabilities which _really_ ought to be integrated into the asdf core: component- specific feature and declaration assertions.
Beyond those use-cases one enters the realm a project like elephant.
Under these circumstances, it would make asdf much more suitable to the purpose for which it is intended if it were changed as follows:
- find-system should be changed. rather than call load and then check whether there was any beneficial effect, it should use its own read- eval loop to interpret .asd files. this loop should - establish a restricted package - disable read evaluation - interpret two forms - asdf:defsystem; in which module and file components only are supported. specialization is declared with the :class attribute - cl:use-package; in which a reference to an unknown package leads to an attempt to load a system of that name. - support a mode which permits :class specifications for unknown classes and initialization arguments other than those from the core classes
Such .asd files would also be loadable as source if separate preparations were made to incorporate the extension modules.
?
--- [0] : http://common-lisp.net/project/elephant/darcs/elephant/ elephant.asd [2] : http://common-lisp.net/project/clbuild/ [ clbuild.lisp and the clbuild shell script are phenomenal for the things they accomplish despite that they really should not need to.] [1] : http://common-lisp.net/project/asdf/manual.html
On Tue, Mar 23, 2010 at 2:35 PM, james anderson james.anderson@setf.dewrote:
[ ... continued ]
It was a simple experiment. Define a restricted package and interpret .asd files with a restricted read-eval loop.
Beautiful. Could you attach this sample patch to the bug report I filed asking for this 'feature'?
Juanjo
On Tue, 23 Mar 2010 14:35:49 +0100, james anderson james.anderson@setf.de wrote:
[ ... continued ]
It was a simple experiment. Define a restricted package and interpret .asd files with a restricted read-eval loop.
For what it's worth, this is what I use in desire, with an obligatory piece of poetry compensating for the inevitable brain damage:
;;;; ;;;; o/~ Below zero, below my need for words o/~ ;;;; o/~ Feel you lifeform, not human. o/~ ;;;; ... ;;;; o/~ Of all the big mistakes I've done o/~ ;;;; o/~ The small ones will remain... o/~ ;;;; (defun invoke-with-safe-reader-context (fn) (let ((*read-eval* nil) (*readtable* (copy-readtable))) (set-dispatch-macro-character ## #. (lambda (stream &optional char sharp) (declare (ignore char sharp)) (let ((*read-suppress* t)) (read stream nil nil t)))) (funcall fn)))
(defmacro with-safe-reader-context (() &body body) `(invoke-with-safe-reader-context (lambda () ,@body)))
(defun map-asd-defsystems (stream fn) (with-safe-reader-context () (flet ((form-defsystem-p (f) (and (consp f) (string= "DEFSYSTEM" (symbol-name (first f))) (or (stringp (second f)) (symbolp (second f)))))) (iter (for pre-read-posn = (file-position stream)) (for form = (handler-case (read stream nil 'das-eof) (serious-condition () ;; seek the offending form (let ((*read-suppress* t)) (file-position stream pre-read-posn) (read stream nil nil))))) (until (eq 'das-eof form)) (when (form-defsystem-p form) (collect (funcall fn form)))))))
(defmacro do-asd-defsystems ((form stream) &body body) `(map-asd-defsystems ,stream (lambda (,form) ,@body)))
(defun normalise-asdf-sysdep (dep) "Given an ASDF system dependency, normalise it by returning the name depended upon as the primary value, and the required version, whenever present as the secondary value." (if (consp dep) (values (second dep) (first dep)) dep))
(defun asdf-system-dependencies (system) "Parse an .asd as if it were declarative." (with-open-file (s (system-definition-pathname system)) (apply #'nconc (do-asd-defsystems (form s) (destructuring-bind (defsystem name &key depends-on &allow-other-keys) form (declare (ignore defsystem)) (when (string-equal name (name system)) (mapcar (compose #'canonicalise-name #'normalise-asdf-sysdep) depends-on)))))))
(defun asdf-hidden-system-names (pathname) "Find out names of ASDF systems hiding in .asd in PATHNAME. A hidden system is a system with a definition residing in a file named differently from that system's name." (let ((primary-system-name (string-upcase (pathname-name pathname)))) (with-open-file (s pathname) (remove nil (do-asd-defsystems (form s) (let ((system-name (string-upcase (string (second form))))) (when (not (string= primary-system-name system-name)) system-name)))))))
-
Daniel Herring -
Faré -
james anderson -
Juan Jose Garcia-Ripoll -
Robert Goldman -
Samium Gromoff -
Tobias C. Rittweiler