[asdf-devel] Package manipulation [ was ASDF traverse changed behavior? ]
On Thu, Mar 18, 2010 at 1:36 AM, Robert Goldman <rpgoldman@sift.info> wrote:
Is it possible that somehow the compilation of asdf-ecl is recording some information about the package that is somehow damaged by the package surgery?
I will dig into that later during the weekend. I think the point is something we discussed long at the ECL mailing list, namely that symbols are NOT created as a compiled file is run. Symbols are constants and as such a compiler is allowed to coalesce all references to symbols in the same compiled file. The problem would thus be the following. The compiler loads the file, it resolves all references to symbols, storing them in the array of constants, including PERFORM. The code is executed. PERFORM is uninterned and a new symbol is created. We reach the DEFMETHOD forms, which now are executed but since they reference the *old* version of the symbol, the method is installed with the wrong name in the wrong generic function. In other words, one can not expect the side effects that operations have in packages work the same for a compiled and a source file. It is just as if you expect that the side effects that aare caused by changing the readtable would also affect the way a cmpiled file behave. Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
On 18 March 2010 03:56, Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com> wrote:
On Thu, Mar 18, 2010 at 1:36 AM, Robert Goldman <rpgoldman@sift.info> wrote:
Is it possible that somehow the compilation of asdf-ecl is recording some information about the package that is somehow damaged by the package surgery?
I will dig into that later during the weekend. I think the point is something we discussed long at the ECL mailing list, namely that symbols are NOT created as a compiled file is run. Symbols are constants and as such a compiler is allowed to coalesce all references to symbols in the same compiled file. The problem would thus be the following. The compiler loads the file, it resolves all references to symbols, storing them in the array of constants, including PERFORM. The code is executed. PERFORM is uninterned and a new symbol is created. We reach the DEFMETHOD forms, which now are executed but since they reference the *old* version of the symbol, the method is installed with the wrong name in the wrong generic function. In other words, one can not expect the side effects that operations have in packages work the same for a compiled and a source file. It is just as if you expect that the side effects that aare caused by changing the readtable would also affect the way a cmpiled file behave.
It may very well be that my recent package manipulations (in hope of achieving hot upgrade of ASDF) are what broke ASDF on ECL, and not Robert's TRAVERSE fixes. I apologize if that is the case. Can you pull 1.651 and see if it fixed the issue? Once again, a test script would be swell. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Never explain. Your friends do not need it and your enemies will never believe you anyway. — Elbert Hubbard
On Thu, Mar 18, 2010 at 4:02 PM, Faré <fahree@gmail.com> wrote:
It may very well be that my recent package manipulations (in hope of achieving hot upgrade of ASDF) are what broke ASDF on ECL, and not Robert's TRAVERSE fixes. I apologize if that is the case. Can you pull 1.651 and see if it fixed the issue?
Sitting behind a firewall, will not be able to pull from git for a few hours. I have a small question though. Why not using the scheme (defpackage :asdf-boot ...) ; all new code using asdf-boot package ; rename package "asdf" -> "asdf-old" ; rename package "asdf-boot" -> "asdf" instead of manually moving the symbols around? The difference here is that the symbols that the compiler reads in "all new code" are well defined and are the same ones that will end up being installed in the final "ASDF" package. Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
On 18 March 2010 11:09, Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com> wrote:
Why not using the scheme
(defpackage :asdf-boot ...) ; all new code using asdf-boot package ; rename package "asdf" -> "asdf-old" ; rename package "asdf-boot" -> "asdf"
instead of manually moving the symbols around? The difference here is that the symbols that the compiler reads in "all new code" are well defined and are the same ones that will end up being installed in the final "ASDF" package.
IIRC, I rejected this plan to allow for update of a running ASDF in a Lisp image where plenty of other packages use ASDF or import its symbols. This packages have to continue working if not recompiled, and they have to work also if recompiled, in which case package renaming could cause some "interesting" warnings. I don't have the final answer on hot upgrade of Lisp software. CL was never intended to solve all the issues of hot code upgrade: there are no code upgrade transactions (and no explicit support for concurrency), no clear internal/external naming strategy (as in Erlang's module-qualified vs unqualified calls), etc. But I think ASDF ought to make a best effort, so that you can at the very least upgrade ASDF when installed as part of Common-Lisp-Controller and similar management layers (that do use package ASDF). [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Life is like an onion: you peel off layer after layer, then you find there is nothing in it.
good afternoon; On 2010-03-18, at 16:23 , Faré wrote:
[ ...]
But I think ASDF ought to make a best effort, so that you can at the very least upgrade ASDF when installed as part of Common-Lisp-Controller and similar management layers (that do use package ASDF).
as a step in that direction, let us enumerate the actions which an upgrade could contemplate and come to terms with the consequences on existing code. in the note below, "not permitted" refers to a situation where the modification must be compatible with a running system and/or binary files, to be loaded as is, without recourse to source files for recompilation. package renaming : is not permitted package deletion : is not permitted package aliasing : is permitted, as long as the name is new symbol export : is not permitted, as the new export can conflict with the content of a using package symbol unexport : is not permitted, as it could eliminate a previously visible symbol symbol uninterning : is not permitted, as it would compromises existing references plain function redefinition as generic : if done with fmakunbound, this is permitted, as only the library definition can exist. generic function redefinition as plain : this is not permitted, as it could purge extension methods. generic function redefinition : via defgeneric is permitted, that is, so long as the lambda list is compatible, as it should not effect extension methods. via fmakunbound is not permitted, as it could purge extension methods. this means that the present (ensure- package :fmakunbound) semantics must be changed to distinguish plain and generic functions. method deletion : as long as the method specializers are asdf classes or common-lisp classes, this is permitted, as this no extension function should fit that constraint. this could be automated independent of generic function redefinition as a means to clean a package of incompatible definitions. (ensure-package :fmakunbound) could use this approach. method addition : same as deletion method redefinition : same as deletion variable rebinding : should be permitted. there are no constants. wrt. methods on the basis of 1.648, there are very few methods which are not already fully specialized and as such amenable to such rules. perform-with-restarts has a default method as the next method for the :around methods. one could argue that this should be distinct base methods, but i don't see a clear reason to do that - especially as it is intended to be _the_ default method. output-files has an :around method which could be specialized. are there other modifications? are there other methods?
On Thu, Mar 18, 2010 at 6:29 PM, james anderson <james.anderson@setf.de>wrote:
On 2010-03-18, at 16:23 , Faré wrote:
[ ...]
But I think ASDF ought to make a best effort, so that you can at the very least upgrade ASDF when installed as part of Common-Lisp-Controller and similar management layers (that do use package ASDF).
as a step in that direction, let us enumerate the actions which an upgrade could contemplate[...]
I have been reading the list of things that can, have and cannot be done and it is way more complex than I had expected. Wouldn't it make sense to split asdf.lisp into two files? asdf-boot.lisp to set up everything _only_ if we need to redefine ASDF functions, classes and other stuff, and asdf.lisp which works as it used to do. In asdf.lisp we may even add a finall call to an optional function (asdf-finish) that polishes the last corners _after_ everything new has been defined. The advantage is that the logic of redefinition and of the ASDF system itself are well separated. It also helps that implementations for which redefinition is not happening (for instance ECL does not load ASDF in the core, it is an optional module), can forget about the complexity of the asdf-boot.lisp part. Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com
: Juanjo
I have been reading the list of things that can, have and cannot be done and it is way more complex than I had expected. Wouldn't it make sense to split asdf.lisp into two files? asdf-boot.lisp to set up everything _only_ if we need to redefine ASDF functions, classes and other stuff, and asdf.lisp which works as it used to do. In asdf.lisp we may even add a finall call to an optional function (asdf-finish) that polishes the last corners _after_ everything new has been defined.
The advantage is that the logic of redefinition and of the ASDF system itself are well separated. It also helps that implementations for which redefinition is not happening (for instance ECL does not load ASDF in the core, it is an optional module), can forget about the complexity of the asdf-boot.lisp part.
I've thought about it. Back before I started work on ASDF 2, but when I needed a known version of ASDF for use with ASDF-DEPENDENCY-GROVEL, I distributed a file no-asdf.lisp with XCVB that deleted the ASDF package and any package that depended on it (recursively), made sure any asdf-module-provider for require was de-registered, and improper *features* removed. It was hard and heavy handed, but it worked. Problem is, this approach works when the loading of files is driven externally by XCVB when compiling from "scratch" (or minimally modified image with, e.g. clc installed), but is wholly unsuitable for ASDF-driven self-upgrade of ASDF. It notably loses the state of currently loaded systems which can cause hours of fun, especially since renaming or deleting packages leads to "interesting" errors when the re-loaded systems will try to redefine their packages. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] My wife and I have clearly defined who's boss. I always take all the important decisions. I decide what is the family's take on the recognition of Communist China, how many B-52's are necessary to the armed forces and what to do about the federal debt. I leave her the petty daily decisions, like determining where we'll live, which school the kids will attend and who we'll have for dinner. — Unknown author, USA, 1950's
: fare : janderson
But I think ASDF ought to make a best effort, so that you can at the very least upgrade ASDF when installed as part of Common-Lisp-Controller and similar management layers (that do use package ASDF).
as a step in that direction, let us enumerate the actions which an upgrade could contemplate and come to terms with the consequences on existing code. in the note below, "not permitted" refers to a situation where the modification must be compatible with a running system and/or binary files, to be loaded as is, without recourse to source files for recompilation.
This deserves to be published in a document. Maybe a page on Cliki. Or a post on a blog, etc.
package renaming : is not permitted package deletion : is not permitted package aliasing : is permitted, as long as the name is new
OK.
symbol export : is not permitted, as the new export can conflict with the content of a using package
symbol unexport : is not permitted, as it could eliminate a previously visible symbol
symbol uninterning : is not permitted, as it would compromises existing references
Hum, yet, we do export new symbols, unexport and unintern old symbols and in practice, it works well enough.
plain function redefinition as generic : if done with fmakunbound, this is permitted, as only the library definition can exist. generic function redefinition as plain : this is not permitted, as it could purge extension methods.
generic function redefinition : via defgeneric is permitted, that is, so long as the lambda list is compatible, as it should not effect extension methods. via fmakunbound is not permitted, as it could purge extension methods. this means that the present (ensure- package :fmakunbound) semantics must be changed to distinguish plain and generic functions.
Sometimes, we do want to purge extension methods. Especially when the extension protocol has changed. As in when the method-combination of some gf was recently obliterated, or simply when the signature of some methods have changed, and you don't want the old methods to be called anymore.
method deletion : as long as the method specializers are asdf classes or common-lisp classes, this is permitted, as this no extension function should fit that constraint. this could be automated independent of generic function redefinition as a means to clean a package of incompatible definitions. (ensure-package :fmakunbound) could use this approach.
method addition : same as deletion
method redefinition : same as deletion
Yes, but much heavier weight than now.
variable rebinding : should be permitted. there are no constants.
Tricky would be symbol being made un-constant or un-special. Would require unintern.
wrt. methods on the basis of 1.648, there are very few methods which are not already fully specialized and as such amenable to such rules.
I don't get it.
perform-with-restarts has a default method as the next method for the :around methods. one could argue that this should be distinct base methods, but i don't see a clear reason to do that - especially as it is intended to be _the_ default method.
Can do. What would that buy us?
output-files has an :around method which could be specialized.
And your conclusion is?
are there other modifications?
are there other methods?
We don't have to deal with every possible corner case that someone could dream of. But we do have to deal with things that actually happened in previous versions of ASDF that people may upgrade from, and corresponding clients that somehow use ASDF. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Death is only a state of mind. Only it doesn't leave you much time to think about anything else.
On 3/18/10 Mar 18 -1:44 PM, Faré wrote:
: fare : janderson
But I think ASDF ought to make a best effort, so that you can at the very least upgrade ASDF when installed as part of Common-Lisp-Controller and similar management layers (that do use package ASDF).
as a step in that direction, let us enumerate the actions which an upgrade could contemplate and come to terms with the consequences on existing code. in the note below, "not permitted" refers to a situation where the modification must be compatible with a running system and/or binary files, to be loaded as is, without recourse to source files for recompilation.
This deserves to be published in a document. Maybe a page on Cliki. Or a post on a blog, etc.
Proposal: let's get something like that written up on Cliki or a blog post by whoever has the energy to do it. I would be willing to formalize it into Texinfo and stuff it into the manual if that's done. Preference: absolute minimum of formatting in the text, so it's easy to translate into Texinfo (texinfo is more "semantic" than things like wiki markup or markdown, so it's tricky for me to use string replacement for much of the translation).
package renaming : is not permitted package deletion : is not permitted package aliasing : is permitted, as long as the name is new
OK.
symbol export : is not permitted, as the new export can conflict with the content of a using package
symbol unexport : is not permitted, as it could eliminate a previously visible symbol
symbol uninterning : is not permitted, as it would compromises existing references
Hum, yet, we do export new symbols, unexport and unintern old symbols and in practice, it works well enough.
I am inclined to agree: how can we not change the set of exports without fully locking ourselves into the existing API? I /do/ agree that such changes should be minimized, though, and presumably should come with instructions about how to handle them in #+asdf<n> code blocks. This is an interesting question --- what should one do if one expects that there might be a downstream incompatible change? Should one recommend that people write code like #+(and asdf2 (not asdf3)) that seems cumbersome....
plain function redefinition as generic : if done with fmakunbound, this is permitted, as only the library definition can exist. generic function redefinition as plain : this is not permitted, as it could purge extension methods.
generic function redefinition : via defgeneric is permitted, that is, so long as the lambda list is compatible, as it should not effect extension methods. via fmakunbound is not permitted, as it could purge extension methods. this means that the present (ensure- package :fmakunbound) semantics must be changed to distinguish plain and generic functions.
Sometimes, we do want to purge extension methods. Especially when the extension protocol has changed. As in when the method-combination of some gf was recently obliterated, or simply when the signature of some methods have changed, and you don't want the old methods to be called anymore.
Presumably it's acceptable if purging extension methods causes an error, since we shouldn't be trying to load an extension that requires methods that no longer exist, right?
method deletion : as long as the method specializers are asdf classes or common-lisp classes, this is permitted, as this no extension function should fit that constraint. this could be automated independent of generic function redefinition as a means to clean a package of incompatible definitions. (ensure-package :fmakunbound) could use this approach.
method addition : same as deletion
method redefinition : same as deletion
Yes, but much heavier weight than now.
variable rebinding : should be permitted. there are no constants.
Tricky would be symbol being made un-constant or un-special. Would require unintern.
Question: if we have only internal constants (i.e., not exported and not for use in extensions), will that be safe? Best, r
On 2010-03-18, at 19:44 , Faré wrote:
: fare : janderson [...]
wrt. methods on the basis of 1.648, there are very few methods which are not already fully specialized and as such amenable to such rules.
I don't get it.
perform-with-restarts has a default method as the next method for the :around methods. one could argue that this should be distinct base methods, but i don't see a clear reason to do that - especially as it is intended to be _the_ default method.
Can do. What would that buy us?
output-files has an :around method which could be specialized.
And your conclusion is?
That there would be no changes required in the asdf implementation itself in order to institute such a rule.
participants (4)
-
Faré -
james anderson -
Juan Jose Garcia-Ripoll -
Robert Goldman