Hi (I'm not sure who is current lead maintainer, but hopefully the group will have some wisdom),
First of all, Happy New Year and thank you to whomever is monitoring this list. Your efforts are highly appreciated.
Our build process for Gendl and GDL for the past several years has involved using monolithic-compile-bundle-op to make concatenated fasls, then loading these fasls into an empty image to create the final build.
We do depend on (and include) uiop, but not asdf, in our final build product.
In order to have uiop available separately for inclusion in the monolithic-compile-bundles, we have been putting a copy of the uiop/ directory (of the same version as the ASDF we're using) into quicklisp/local-projects/.
Some time between ASDF 3.2.0.1 and ASDF 3.3.1, the function check-not-old-asdf-system got modified to add "uiop" to the test (first line of the function). This results (in ASDF 3.3.1) in a same-version uiop not getting loaded at all into a monolithic-compile-bundle, even if the system of said monolithic-compile-bundle contains a dependency on uiop and uiop is available in quicklisp/local-projects/.
For this reason, out-of-the-box ASDF 3.3.1 was not working for our build process. I have "fixed" the problem temporarily by modifying my local asdf.lisp to remove the "uiop" from the test, i.e. I've replaced
(or (not (member name '("asdf" "uiop") :test 'equal)) ....
with
(or (not (member name '("asdf") :test 'equal)) ...
as the first line of the defun check-not-old-asdf-system.
Question: Is there a supported way to do what I'm trying to do, without modifying the ASDF source code? Intuitively, it seems to me that we should be able to include uiop in build products using monolithic-compile-bundle-op, without including asdf.
On 12 Jan 2018, at 22:00, Dave Cooper wrote:
Hi (I'm not sure who is current lead maintainer, but hopefully the group will have some wisdom),
I'm still the lead maintainer for the nonce. I believe Faré would like to see a replacement who is able to give more time and energy to the task. For that matter, I would, too! But no one has stepped up yet.
First of all, Happy New Year and thank you to whomever is monitoring this list. Your efforts are highly appreciated.
Our build process for Gendl and GDL for the past several years has involved using monolithic-compile-bundle-op to make concatenated fasls, then loading these fasls into an empty image to create the final build.
We do depend on (and include) uiop, but not asdf, in our final build product.
In order to have uiop available separately for inclusion in the monolithic-compile-bundles, we have been putting a copy of the uiop/ directory (of the same version as the ASDF we're using) into quicklisp/local-projects/.
Some time between ASDF 3.2.0.1 and ASDF 3.3.1, the function check-not-old-asdf-system got modified to add "uiop" to the test (first line of the function). This results (in ASDF 3.3.1) in a same-version uiop not getting loaded at all into a monolithic-compile-bundle, even if the system of said monolithic-compile-bundle contains a dependency on uiop and uiop is available in quicklisp/local-projects/.
For this reason, out-of-the-box ASDF 3.3.1 was not working for our build process. I have "fixed" the problem temporarily by modifying my local asdf.lisp to remove the "uiop" from the test, i.e. I've replaced
(or (not (member name '("asdf" "uiop") :test 'equal)) ....
with
(or (not (member name '("asdf") :test 'equal)) ...
as the first line of the defun check-not-old-asdf-system.
Question: Is there a supported way to do what I'm trying to do, without modifying the ASDF source code? Intuitively, it seems to me that we should be able to include uiop in build products using monolithic-compile-bundle-op, without including asdf.
I am afraid I don't have much light to shine here. It seems to me that we have an issue here that comes from using ASDF for two different purposes. In the case of what I think of as "normal" ASDF usage, ASDF ensures that a currently running lisp image gets into a consistent state with respect to software systems, and can be kept in (or perhaps restored to) such a state as source code is modified.
The bundle operations do something that is substantially different. They are being used to *construct* a new and different Lisp image, I believe, and so comparing to the state of the running image may be wrong. This seems like such a case: the state of the ASDF system and UIOP in the *running* image doesn't necessarily correspond to the state of those systems in the image you are constructing. In that case, it seems to me that check-not-old-asdf-system may be simply inappropriate as a check in some (all?) bundle operations. But I would be hard pressed to say when it is and is not appropriate. E.g., presumably it is appropriate in image building, since any image one builds would include the current running ASDF. But that argument does not seem to hold for bundles full of fasls or source code, does it?
I wonder if it is possible that we should move this check out of its current location, where it attempts to hide out-of-date versions of ASDF and UIOP from view. Maybe we should instead make an installed version of ASDF visible to `locate-system` so that any out-of-date version will be shadowed that way, instead of needing to be hidden from `locate-system`, as it is now.
In one sense at least, Faré is right that I should be replaced as maintainer. I have never used any of the bundle operations, and feel ambivalent about their inclusion into ASDF. They crept in as a side-effect of supporting lisps that use the C compiler, but they are a *substantial* increase in the breadth and maintenance difficulty of ASDF. I'm afraid I have neither the time nor the interest to carry that burden. I would be happy to yield it to others.
: Dave Cooper
: Robert Goldman rpgoldman@sift.info
Summarizing: * gendl (and gdl with it) uses monolithic-compile-bundle-op, depends-on uiop, not asdf. * gendl tries to use uiop from source to include it without asdf, but asdf 3.3 broke that.
My understanding of the reason for this breakage is as follows: * to ensure safe upgrade, asdf specially loads itself as the start of any call to operate (otherwise, a dependency on asdf could cause an upgrade in the middle, which would totally break everything.) To prevent catastrophic downgrade, asdf specially ignores asdf.asd unless its version is at least as recent as the current one. * asdf has a special relationship to uiop, because it kind of depends on uiop, but at least since 3.3.0, asdf.asd couldn't depend on uiop.asd because the above rule could then cause circular dependencies. asdf.asd instead has a magic asdf/driver system that magically mimic what is in uiop.asd. * To prevent potentially conflicting redefinition within the same build, asdf 3.3 therefore specially ignores uiop.asd unless its version is *strictly* superior to what is built into asdf.
Therefore, if you want to force asdf 3.3.x to load uiop, I recommend patching uiop/version.lisp to append appending .0.1 to the *uiop-version* variable value.
Sorry for the complication, but that's the simplest I could come up with. If you can think of a better arrangement, I'm sure the next ASDF powers that be will be listening to you.
I recommend against changing the logic of ASDF.
Question: Is there a supported way to do what I'm trying to do, without modifying the ASDF source code? Intuitively, it seems to me that we should be able to include uiop in build products using monolithic-compile-bundle-op, without including asdf.
I would say that gendl is large enough that the 6000 lines of asdf minus uiop shouldn't be a big burden; but if you insist on shipping with uiop and not asdf, I recommend bumping oh-so-slightly the version of uiop.
:Robert In that case, it seems to me that check-not-old-asdf-system may be simply inappropriate as a check in some (all?) bundle operations. But I would be hard pressed to say when it is and is not appropriate. E.g., presumably it is appropriate in image building, since any image one builds would include the current running ASDF. But that argument does not seem to hold for bundles full of fasls or source code, does it?
Well, since at least on most implementations, building the fasls involve loading the code in the current image, there isn't much leeway by which asdf could afford not to use the same check-not-old-asdf-system function. A future version of ASDF that seriously supports cross-compilation might, but we're not quite there yet (see the TODO for hints, if you're interested in making it happen).
In one sense at least, Faré is right that I should be replaced as maintainer. I have never used any of the bundle operations, and feel ambivalent about their inclusion into ASDF. They crept in as a side-effect of supporting lisps that use the C compiler, but they are a substantial increase in the breadth and maintenance difficulty of ASDF. I'm afraid I have neither the time nor the interest to carry that burden. I would be happy to yield it to others.
There certainly was a mission creep in ASDF 3 (see my 2014 article for some explanations). I don't regret making it happen, even though bugfixing lasted until 2017. The code should be pretty stable now — though, who knows for sure?
Whatever I or other people wish, whoever it is who does the grunt work that others won't do shan't be blamed by those who are not willing to do (or fund) said grunt work.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If normative judgments cannot be rational, then science is actually useless. — François Guillaumat
On 16 Jan 2018, at 14:17, Faré wrote:
:Robert In that case, it seems to me that check-not-old-asdf-system may be simply inappropriate as a check in some (all?) bundle operations. But I would be hard pressed to say when it is and is not appropriate. E.g., presumably it is appropriate in image building, since any image one builds would include the current running ASDF. But that argument does not seem to hold for bundles full of fasls or source code, does it?
Well, since at least on most implementations, building the fasls involve loading the code in the current image, there isn't much leeway by which asdf could afford not to use the same check-not-old-asdf-system function. A future version of ASDF that seriously supports cross-compilation might, but we're not quite there yet (see the TODO for hints, if you're interested in making it happen).
Am I correct in thinking that Dave's way of building monolithic bundles of either fasls or source code are, at least potentially, a baby version of cross-compilation? It seems like these are interesting *specifically* because they could be loaded into *different* images (otherwise, it's not clear to me why it would be better to build a monolithic FASL than just build an image).
In that case, since this would effectively be cross-compilation (albeit a trivial case of it), it's not surprising that the logic for dealing with built-in dependencies like ASDF can go awry.
In which case delivering with Docker might be the better approach! ;-)
Best, r
:Robert Am I correct in thinking that Dave's way of building monolithic bundles of either fasls or source code are, at least potentially, a baby version of cross-compilation? It seems like these are interesting specifically because they could be loaded into different images (otherwise, it's not clear to me why it would be better to build a monolithic FASL than just build an image).
Indeed, (monolithic) bundles are compiled in one image, loaded in another. Same implementation, but different image. But then, as much may apply to regular fasls; and this matters when e.g. some #+ compilation happens based on user-configured features.
In that case, since this would effectively be cross-compilation (albeit a trivial case of it), it's not surprising that the logic for dealing with built-in dependencies like ASDF can go awry.
Indeed.
In which case delivering with Docker might be the better approach! ;-)
Did you mean Bazel? :-)
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org What is the shortest joke? — Communism What is the longest joke? — 70 years of Communism in Russia
-
Dave Cooper -
Faré -
Robert Goldman