I've been reading the manual, papers, and slides on ASDF and XCVB. As a frame of reference, I prefer the qualities exhibited by Racket's implementation of modules. I was disappointed both to see XCVB bitrot, and then to see that a major ASDF overhaul would be necessary.
That said, ignoring Racket/ASDF/XCVB, I'm curious about how loading multiple versions of the same system could possibly be implemented in Common Lisp (CL). I'm sure people have thought about it, but I'm having a hard time finding references. I'm afraid I've been skimming some of the material linked off of the ASDF and XCVB pages, so forgive me if I missed something.
On the JVM, there's the notion of "classloader" that can be used to load multiple versions of the same class. Unfortunately, the only approach I can imagine is to use some form of package renaming. There appear to be several variations floating around [1], and the trade-offs aren't clear to me.
If I think about how it might be done in ASDF, it seems like it would require a series of things:
* intercept package definition to rename with version * :around-compile to handle some kind of per-system aliasing so code doesn't have to change * teaching dependency resolution how to find the renamed, versioned packages/systems
But that's a wild guess. And the more I think about it, I wonder what to do about some code that uses strings to put together a symbol - I don't think symbol-macrolet can help there.
[1] implementation-specific package-local-nicknames, :around-compile in ASDF, "pseudonyms"
Maybe, under rare circumstances. With my QA hat on, I say better not to try.
I've been reading the manual, papers, and slides on ASDF and XCVB. As a frame of reference, I prefer the qualities exhibited by Racket's implementation of modules. I was disappointed both to see XCVB bitrot, and then to see that a major ASDF overhaul would be necessary.
That said, ignoring Racket/ASDF/XCVB, I'm curious about how loading multiple versions of the same system could possibly be implemented in Common Lisp (CL). I'm sure people have thought about it, but I'm having a hard time finding references. I'm afraid I've been skimming some of the material linked off of the ASDF and XCVB pages, so forgive me if I missed something.
On the JVM, there's the notion of "classloader" that can be used to load multiple versions of the same class. Unfortunately, the only approach I can imagine is to use some form of package renaming. There appear to be several variations floating around [1], and the trade- offs aren't clear to me.
If I think about how it might be done in ASDF, it seems like it would require a series of things:
- intercept package definition to rename with version
- :around-compile to handle some kind of per-system aliasing so code doesn't have to change
- teaching dependency resolution how to find the renamed, versioned packages/systems
But that's a wild guess. And the more I think about it, I wonder what to do about some code that uses strings to put together a symbol - I don't think symbol-macrolet can help there.
[1] implementation-specific package-local-nicknames, :around-compile in ASDF, "pseudonyms"
-- Ian Tegebo
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On Thu, Apr 28, 2016 at 8:05 PM, Ian Tegebo ian.tegebo@gmail.com wrote:
I've been reading the manual, papers, and slides on ASDF and XCVB. As a frame of reference, I prefer the qualities exhibited by Racket's implementation of modules. I was disappointed both to see XCVB bitrot, and then to see that a major ASDF overhaul would be necessary.
Well, unhappily CL is stuck in the early 1990s and unlikely to grow any Racket-like features. As for XCVB, it had many good ideas, but some of them in retrospect were adding too much complexity for not enough benefits. If you want to build a Lisp-specific build system today, ASDF3 might be a better place to start hacking than XCVB: a few of XCVB's good ideas were added to ASDF2 and ASDF3, and a few of them are in the asdf/TODO in case anyone wants to write an ASDF4.
One issue with XCVB is that since it was not backward-compatible with ASDF, it had to be ten times better than ASDF on every implementation on every OS for every system before people would bother switching. Or you'd have to convert every system to the new build. Bazel now supports CL, and is also incompatible and requires users to convert every system they use. Maybe it's better enough, but I don't think so, and certainly not for all Lisp implementations at this point, only SBCL on Linux (and soon MacOS X?). See https://github.com/qitab/bazelisp
As to me, if I had to design a build system today... see http://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems/
That said, ignoring Racket/ASDF/XCVB, I'm curious about how loading multiple versions of the same system could possibly be implemented in Common Lisp (CL). I'm sure people have thought about it, but I'm having a hard time finding references. I'm afraid I've been skimming some of the material linked off of the ASDF and XCVB pages, so forgive me if I missed something.
What is your use case? Do you want a fully automated general answer, or just a manual hack that works for one or a few very specific systems on a specific implementation? In the former case, you're facing an uphill battle. In the latter case, many hacks can help you.
Various hacks:
* unregistering a system with clear-system then calling initialize-source-registry with a different value so you see one system at one time then the other system at another time. * playing games with ASDF plan generation to load all the dependencies of a system without loading the system itself. * detecting packages present before and after loading the system itself, so you can rename them away. * using my system package-renaming to cleverly rename relevant packages around the compilation, loading and/or use of one system. * fixing your systems to not refer to packages by name at run-time, and if possible not at load-time either (or make sure to wrapper load-time in proper package-renaming).
On the JVM, there's the notion of "classloader" that can be used to load multiple versions of the same class. Unfortunately, the only approach I can imagine is to use some form of package renaming. There appear to be several variations floating around [1], and the trade-offs aren't clear to me.
If I think about how it might be done in ASDF, it seems like it would require a series of things:
- intercept package definition to rename with version
- :around-compile to handle some kind of per-system aliasing so code doesn't
have to change
- teaching dependency resolution how to find the renamed, versioned
packages/systems
But that's a wild guess. And the more I think about it, I wonder what to do about some code that uses strings to put together a symbol - I don't think symbol-macrolet can help there.
[1] implementation-specific package-local-nicknames, :around-compile in ASDF, "pseudonyms"
If what you want is a general way to have multiple versions of any system in a same image, you're out of luck, and/or you need to use a CL-in-CL implementation (or CL-in-foo) that shields the various instances of user code from each other.
If what you want is a build system that is full-featured like XCVB with plenty of dependencies yet compiles things in-image like ASDF but using user-provided versions of the same systems that your build system depends on, then yes, some magic package renaming could help, and could be a one-time thing if your build system never calls a function that assumes unrenamed packages at runtime. Or then again, you could use the XCVB farmer model, where the build image with lots of dependencies farms the compilation to workers that only need to load a small stub, and no package renaming is necessary. If your workers also support forking, this can be very efficient, too.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Computer Science is no more about computers than astronomy is about telescopes. — E. W. Dijkstra
Thanks for the detailed response. Comments inline,
On Fri, Apr 29, 2016 at 2:23 AM, Faré fahree@gmail.com wrote:
On Thu, Apr 28, 2016 at 8:05 PM, Ian Tegebo ian.tegebo@gmail.com wrote:
I've been reading the manual, papers, and slides on ASDF and XCVB. As a frame of reference, I prefer the qualities exhibited by Racket's implementation of modules. I was disappointed both to see XCVB bitrot,
and
then to see that a major ASDF overhaul would be necessary.
[...] As to me, if I had to design a build system today... see http://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems/
I've booked travel to the Land of the Houyhnhnms , but have yet to depart.
That said, ignoring Racket/ASDF/XCVB, I'm curious about how loading multiple
versions of the same system could possibly be implemented in Common Lisp (CL). [...]
What is your use case? Do you want a fully automated general answer, or just a manual hack that works for one or a few very specific systems on a specific implementation? In the former case, you're facing an uphill battle. In the latter case, many hacks can help you.
The former is intriguing, the latter's useful. Curiosity about a general solution is what prompted this thread, but I'm happy to learn of workarounds.
Various hacks:
[...]
- detecting packages present before and after loading the system
itself, so you can rename them away.
- using my system package-renaming to cleverly rename relevant
packages around the compilation, loading and/or use of one system.
- fixing your systems to not refer to packages by name at run-time,
and if possible not at load-time either (or make sure to wrapper load-time in proper package-renaming).
With respect to package renaming, I've asked Michał Herda to comment on whatever issues he'd had with "destructive package renaming" [1]. I understand it as referring to the state-based global-reader implementation of packages, which seems to entail well known problems like reentrancy and readtable leakage.
Would you correct me or otherwise expand on the matter? I.e. I'm trying to understand the tradeoffs between package renaming methods. My use cases are:
a) package local nicknames for readability b) implementing a variant of Racket's "#lang" c) supporting multiple versions of a dependency
I know that some implementations directly support a), and that both a) and b) may be implemented with ASDF, but I didn't know about c).
On the JVM, there's the notion of "classloader" that can be used to load
multiple versions of the same class. Unfortunately, the only approach I
can
imagine is to use some form of package renaming.
If what you want is a general way to have multiple versions of any system in a same image, you're out of luck, and/or [...]
Yes, I'm primarily interested in a general way: thanks for entertaining the discussion.
If what you want is a build system that is full-featured like XCVB
with plenty of dependencies yet compiles things in-image like ASDF [...]
Or then again,
you could use the XCVB farmer model, where the build image with lots of dependencies farms the compilation to workers that only need to load a small stub, and no package renaming is necessary.
This approach sounds like one motivated by version conflict between the build system and system being built. Is that correct?
I wasn't really thinking about that, but now that I am I'm reminded of various parts of your paper [2]. The bootstrapping, upgrading, and distribution of ASDF seems very thorny indeed.
[1]: https://github.com/phoe-krk/pseudonyms/issues/2#issuecomment-215534567 [2]: http://fare.tunes.org/files/asdf3/asdf3-2014.html
Curiosity about a general solution is what prompted this thread,
Oh well, I'm not going to expect much to come out of it, then.
With respect to package renaming, I've asked Michał Herda to comment on whatever issues he'd had with "destructive package renaming" [1]. I understand it as referring to the state-based global-reader implementation of packages, which seems to entail well known problems like reentrancy and readtable leakage.
No, I use rename-package, but I leave it as the programmer's responsibility to ensure that a package's canonical name while compiling a fasl will be a valid name for the correct package at load time. In the general case, this might entail instrumentation or shadowing of defpackage and make-package. No reader hack necessary.
[The XCVB] approach sounds like one motivated by version conflict between the build system and system being built. Is that correct?
Yes. Having lots of dependencies means that "interesting" things may happen if you start compiling an incompatible version of a library you're using. Or even a compatible version that resets important data structures in one file that will only be filled in in another.
I wasn't really thinking about that, but now that I am I'm reminded of various parts of your paper [2]. The bootstrapping, upgrading, and distribution of ASDF seems very thorny indeed.
Indeed. Thanks for being a good public.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org A president worth voting for wouldn't run for office.
Eran Gat, if I remember correctly, had an implementation of lexically scoped packages. That would provide the capability for having multiple versions of the same system live in one lisp image, but would require rewriting the systems in question and their consumers.
TBH, this seems to me like a pretty esoteric use case: I'm curious about how and when you encounter it.
If you're interested in cool language features, that's one thing, but I can't imagine as an engineering matter that you couldn't hack around the few times this would happen.
Note that there's a less demanding item -- having ASDF be able to manage a set of libraries with multiple versions, without being able to load different versions of one library into a single lisp image. This might occur more frequently -- e.g. I could imagine systems that are bug-compatible with different versions of CFFI.
Cheers
On Apr 28, 2016, at 19:05, Ian Tegebo ian.tegebo@gmail.com wrote:
I've been reading the manual, papers, and slides on ASDF and XCVB. As a frame of reference, I prefer the qualities exhibited by Racket's implementation of modules. I was disappointed both to see XCVB bitrot, and then to see that a major ASDF overhaul would be necessary.
That said, ignoring Racket/ASDF/XCVB, I'm curious about how loading multiple versions of the same system could possibly be implemented in Common Lisp (CL). I'm sure people have thought about it, but I'm having a hard time finding references. I'm afraid I've been skimming some of the material linked off of the ASDF and XCVB pages, so forgive me if I missed something.
On the JVM, there's the notion of "classloader" that can be used to load multiple versions of the same class. Unfortunately, the only approach I can imagine is to use some form of package renaming. There appear to be several variations floating around [1], and the trade-offs aren't clear to me.
If I think about how it might be done in ASDF, it seems like it would require a series of things:
- intercept package definition to rename with version
- :around-compile to handle some kind of per-system aliasing so code doesn't have to change
- teaching dependency resolution how to find the renamed, versioned packages/systems
But that's a wild guess. And the more I think about it, I wonder what to do about some code that uses strings to put together a symbol - I don't think symbol-macrolet can help there.
[1] implementation-specific package-local-nicknames, :around-compile in ASDF, "pseudonyms"
-- Ian Tegebo
On Fri, Apr 29, 2016 at 6:10 AM, Robert P. Goldman rpgoldman@sift.net wrote:
TBH, this seems to me like a pretty esoteric use case: I'm curious about how and when you encounter it.
Elsewhere, it happens in large projects where two direct dependencies themselves depend on different versions of the same library. I haven't had this happen to me in CL, but I expect it to happen if exact version matching were to become prevalent.
In reading this list, and elsewhere, I understand that versioning becomes controversial. I've experienced both sides of the problem where you either under or over specify the versions of dependencies. Exact version matching is a useful tool to limit the impact of upstream changes. However, it does induce trivial domino-updates. At Amazon, I got used to some tooling and process that reduces the overhead, while also seeing the turmoil caused by teams having to support old versions (they'd pinned/used-exact and would never have "the resources" to update).
If you're interested in cool language features, that's one thing, but I can't imagine as an engineering matter that you couldn't hack around the few times this would happen.
While I am interested in cool [features] for HOT languages [1], I recognize the pragmatic approach is sort out the upstream divergence and find some workaround - I'm not arguing one way or the other - just wondered whether and how one might sort it out at the system-builder layer.
[1]: http://dl.acm.org/citation.cfm?id=277730
On Fri, Apr 29, 2016 at 3:26 PM, Ian Tegebo ian.tegebo@gmail.com wrote:
Elsewhere, it happens in large projects where two direct dependencies themselves depend on different versions of the same library. I haven't had this happen to me in CL, but I expect it to happen if exact version matching were to become prevalent.
In reading this list, and elsewhere, I understand that versioning becomes controversial. I've experienced both sides of the problem where you either under or over specify the versions of dependencies. Exact version matching is a useful tool to limit the impact of upstream changes. However, it does induce trivial domino-updates. At Amazon, I got used to some tooling and process that reduces the overhead, while also seeing the turmoil caused by teams having to support old versions (they'd pinned/used-exact and would never have "the resources" to update).
This is called "DLL Hell". Don't go there. If you go there, use tools like NixOS that can help you survive.
ASDF just isn't the right tool to attempt things like that, and is unlikely to ever grow into one.
If you're interested in cool language features, that's one thing, but I can't imagine as an engineering matter that you couldn't hack around the few times this would happen.
While I am interested in cool [features] for HOT languages [1], I recognize the pragmatic approach is sort out the upstream divergence and find some workaround - I'm not arguing one way or the other - just wondered whether and how one might sort it out at the system-builder layer.
I like Racket units. They were one of the inspirations for lisp-interface-library. https://github.com/fare/lisp-interface-library
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If it's not worth doing right, it's not worth doing. — Scott McKay
-
Faré -
Ian Tegebo -
Robert P. Goldman -
Stelian Ionescu