Anyone interested in "package versioning"?
Hi CL Pros, I just came across a post on Quora <https://www.quora.com/Where-did-we-go-wrong-Why-didnt-Common-Lisp-fix-the-world/answer/Robert-Smith-9?srid=dnzK> by Robert Smith where he mentions the idea of assigning some sort of "version" to Common Lisp packages in order to improve code configuration control. This seems to me to be a pretty neat and interesting idea! Has anyone of you explored such a concept, or know of anyone that did? Or, is anyone of you curious about it? I think pretty seriously that I will implement something like this as a CL extension in my upcoming MKCL 2.0. Cheers, Jean-Claude Beaudoin
On Wed, May 18, 2016 at 12:21 AM, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
Hi CL Pros,
I just came across a post on Quora by Robert Smith where he mentions the idea of assigning some sort of "version" to Common Lisp packages in order to improve code configuration control.
This seems to me to be a pretty neat and interesting idea!
Has anyone of you explored such a concept, or know of anyone that did? Or, is anyone of you curious about it?
I think pretty seriously that I will implement something like this as a CL extension in my upcoming MKCL 2.0.
ASDF already has some notion of system version. There were some discussions on the list on how to make it extensible, if you care for more than its "semantic versioning". But really, if you really care about configuration control, what you need is a whole-world build system like NixOS or Bazel. No amount of hacking ASDF will take you remotely near there. See also chapter 9 of my Houyhnhnm saga: http://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems/ —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Uniquely in us, nature opens her eyes and sees that she exists. — Raymond Tallis (1946–)
On Wed, May 18, 2016 at 1:19 AM, Faré <fahree@gmail.com> wrote:
On Wed, May 18, 2016 at 12:21 AM, Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com> wrote:
Hi CL Pros,
I just came across a post on Quora by Robert Smith where he mentions the idea of assigning some sort of "version" to Common Lisp packages in
order to
improve code configuration control.
This seems to me to be a pretty neat and interesting idea!
Has anyone of you explored such a concept, or know of anyone that did? Or, is anyone of you curious about it?
I think pretty seriously that I will implement something like this as a CL extension in my upcoming MKCL 2.0.
ASDF already has some notion of system version. There were some discussions on the list on how to make it extensible, if you care for more than its "semantic versioning".
But really, if you really care about configuration control, what you need is a whole-world build system like NixOS or Bazel. No amount of hacking ASDF will take you remotely near there.
See also chapter 9 of my Houyhnhnm saga: http://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems/
Interesting material to consider that you point to here. Thank you Faré.
On Wed, 2016-05-18 at 00:21 -0400, Jean-Claude Beaudoin wrote:
Hi CL Pros,
I just came across a post on Quora <https://www.quora.com/Where-did-we-go-wrong-Why-didnt-Common-Lisp-fi x-the-world/answer/Robert-Smith-9?srid=dnzK> by Robert Smith where he mentions the idea of assigning some sort of "version" to Common Lisp packages in order to improve code configuration control.
This seems to me to be a pretty neat and interesting idea!
Has anyone of you explored such a concept, or know of anyone that did? Or, is anyone of you curious about it?
It's a very bad idea to do code versioning in the linker/loader in order to load two versions at the same time. Saying it makes code-reuse hard is preposterous. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Can you elaborate on that? The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names. I think it *is* already possible, even if no convenient machinery for it exists, to load the same symbols under different packages (i.e. to use different names for different versions of the same concepts). (defpackage p ...) (load (compile-file "p-1.0.lisp")) (rename-package "P" "P-1.0") (defpackage p ...) (load (compile-file "p-1.1.lisp")) (rename-package "P" "P" "P-1.1") p-*.lisp start with (in-package :p) Then you can compile code against p (latest version), p-1.0 and/or p-1.1. Of course each p-*.lisp must be "nice" - only define things in package p. On 18 May 2016 at 11:36, Stelian Ionescu <sionescu@cddr.org> wrote:
On Wed, 2016-05-18 at 00:21 -0400, Jean-Claude Beaudoin wrote:
Hi CL Pros,
I just came across a post on Quora <https://www.quora.com/Where-did-we-go-wrong-Why-didnt-Common-Lisp-fi x-the-world/answer/Robert-Smith-9?srid=dnzK> by Robert Smith where he mentions the idea of assigning some sort of "version" to Common Lisp packages in order to improve code configuration control.
This seems to me to be a pretty neat and interesting idea!
Has anyone of you explored such a concept, or know of anyone that did? Or, is anyone of you curious about it?
It's a very bad idea to do code versioning in the linker/loader in order to load two versions at the same time. Saying it makes code-reuse hard is preposterous.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On 18 May 2016, at 13:57, Alessio Stalla <alessiostalla@gmail.com> wrote:
Can you elaborate on that?
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Indeed. However there are places where having in thw same image multiple versions of the "same" component may be useful. Eg. an application that wants to load older versions of file formats or database structures. There is also the case of library components, when you want or need to load different components depending on different versions in the same image. A depends on B and C. B depends on L version 1, C depends on L version 2. Would being able to deal with such situations be better than the current situation where basically Xach ensures that all the libraries published in quicklisp can compile on sbcl using the same version of all components?
I think it *is* already possible, even if no convenient machinery for it exists, to load the same symbols under different packages (i.e. to use different names for different versions of the same concepts).
(defpackage p ...) (load (compile-file "p-1.0.lisp")) (rename-package "P" "P-1.0") (defpackage p ...) (load (compile-file "p-1.1.lisp")) (rename-package "P" "P" "P-1.1")
p-*.lisp start with (in-package :p)
Then you can compile code against p (latest version), p-1.0 and/or p-1.1. Of course each p-*.lisp must be "nice" - only define things in package p.
Or conversely the original name of the package could contain a major version number (then I'd advocate against providing a versionless nickname, given that a change of major indicates a change of API).
On 18 May 2016 at 11:36, Stelian Ionescu <sionescu@cddr.org> wrote: On Wed, 2016-05-18 at 00:21 -0400, Jean-Claude Beaudoin wrote:
Hi CL Pros,
I just came across a post on Quora <https://www.quora.com/Where-did-we-go-wrong-Why-didnt-Common-Lisp-fi
Don't worry too much about this article, it falls into the "Social Problem Of Lisp" category. Useless.
x-the-world/answer/Robert-Smith-9?srid=dnzK> by Robert Smith where he mentions the idea of assigning some sort of "version" to Common Lisp packages in order to improve code configuration control.
This seems to me to be a pretty neat and interesting idea!
Has anyone of you explored such a concept, or know of anyone that did? Or, is anyone of you curious about it?
It's a very bad idea to do code versioning in the linker/loader in order to load two versions at the same time. Saying it makes code-reuse hard is preposterous.
While unix systems can select amongst different versions of libraries at link time, they don't allow linking with different versions at the same time by the same component. An application would have to do it explicitely with dlopen/dlsym.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
-- __Pascal Bourguignon__
On Wed, May 18, 2016 at 7:57 AM, Alessio Stalla <alessiostalla@gmail.com> wrote:
Can you elaborate on that?
Not really yet. I haven't though enough about it at this point, the idea is still too fresh. But I will at least mention my main source of inspiration on the subject, coming from my Smalltalk days (of somewhat fading memory): the ENVY/Developer system.
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Well, "packages" seem to me to be the only system structuring device offered by the CL spec. So you have to work from there toward something more appropriate, I'd say. In a somewhat upward compatible manner I'd hope.
I think it *is* already possible, even if no convenient machinery for it exists, to load the same symbols under different packages (i.e. to use different names for different versions of the same concepts).
(defpackage p ...) (load (compile-file "p-1.0.lisp")) (rename-package "P" "P-1.0") (defpackage p ...) (load (compile-file "p-1.1.lisp")) (rename-package "P" "P" "P-1.1")
p-*.lisp start with (in-package :p)
Then you can compile code against p (latest version), p-1.0 and/or p-1.1. Of course each p-*.lisp must be "nice" - only define things in package p.
This is an interesting line of argument you've got here... Thanks,
: Alessio Stalla <alessiostalla@gmail.com> : Jean-Claude Beaudoin <jean.claude.beaudoin@gmail.com>
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Excellently summarized.
Well, "packages" seem to me to be the only system structuring device offered by the CL spec. So you have to work from there toward something more appropriate, I'd say. In a somewhat upward compatible manner I'd hope.
When your only hammer is CL packages, expect a lot of hurt fingers. You may also be confusing many different issues: Q1: What are the current best practices when programming in CL on top of the existing package infrastructure? A1: Avoid sharing packages between several sytems. Use one or multiple packages per system, maybe even one or more packages per file, but to avoid modifying other systems' packages, except through well-defined APIs. But in practice such APIs do exist, or are defined post-hoc, and the build system mostly doesn't and cannot know about them. Q2: How do I replace packages with something better, and implement it on top of CL? A2: Look at what languages with advanced module systems are doing, e.g. Racket. But don't expect interoperation between such code and regular CL code to be seamless; your new language can easily FFI into CL, but you'll need tricks akin to extern "C" { ... } to specify entry points for CL code to call into your extended language. But is vanilla CL the correct starting point to develop such a new language? Q3: How do I replace packages with something better, and implement CL on top of that? A3: As with A2, look at a clueful language such as Racket; and again, interoperation will not be seamless. And then if what you implement is bug-compatible with packages and the rest of CL, you'll be able to run CL code on top of your language. But is that what you want? In all these cases, the idea of loading multiple versions of a package unmodified in a same image makes no sense whatsoever. Just not possible as a general mechanism. Sure, by manually editing the code, you can fork a system and rename some packages. Do it, if that's what you need to do. That is NOT AT ALL the same thing as "loading multiple versions of a package in a same image". Now if you have a virtualized implementation of CL (e.g. use multiple CL processes), then yes, it's trivial to have these many images each load its own package. Do it, too, if that's what you need.
I think it *is* already possible, even if no convenient machinery for it exists, to load the same symbols under different packages (i.e. to use different names for different versions of the same concepts).
(defpackage p ...) (load (compile-file "p-1.0.lisp")) (rename-package "P" "P-1.0") (defpackage p ...) (load (compile-file "p-1.1.lisp")) (rename-package "P" "P" "P-1.1")
p-*.lisp start with (in-package :p)
Then you can compile code against p (latest version), p-1.0 and/or p-1.1. Of course each p-*.lisp must be "nice" - only define things in package p.
This is an interesting line of argument you've got here...
If the p-*.lisp are _actually_ nice, you don't need any such brain damage. So keep your code nice, so that no one needs to such stupid evil things doomed to fail. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Foolishness is rarely a matter of lack of intelligence or even lack of information. — John McCarthy
On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <alessiostalla@gmail.com> wrote:
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
I thought I might as well clarify what I meant, since I wrote it. Systems are what should get versioned but systems are not what are referred to by source files. Systems largely coordinate the compilation and loading of files. Source files generally have no notion of a system, actually, which may be what is ultimately problematic. Packages are referred to by source files, however. When I said packages, I meant it, and it was not a conflation with the notion of a system. At the companies I have worked in, we rake in 100s of systems, which consist of n*100s of systems where n is the average number of packages per system. These version collisions happen enough, and we are lucky when they're not silent. Often enough, it's an extremely subtle issue that system A depends on {system B at time T1} and system C depends on {system B at time T2}, where T1 and T2 correspond to different versions of system B, even if not explicitly labeled as such. Let's suppose we simplify the problem however and assume that B has been properly versioned in time, and systems A and C refer properly to the correct version of system B as a dependency. Even then, they will not be able to be simultaneously loaded in a functionally correct fashion. (They *can* both be loaded, but you'll do some clobbering of state along the way.) Whether we think about modules as systems or as packages or as whatever, the problem would need to be addressed at the package level, if we are not assuming that we can't synchronize the universe of people writing Common Lisp. Overall, I'd summarize as this. Whether we have the technology in the Common Lisp ecosystem to accommodate this problem or not, it's my opinion and the opinion of many of my colleagues that there is a problem for development teams making extensive use of open and closed source code. There definitely is not a discipline that you see with other languages and their ecosystems (JS/npm, Clojure, etc.) about proper versioning of systems and furthermore proper loading of the proper versions. I don't think that's debatable. The closest we have to this solution is relying on the global nature of a particular Quicklisp distribution, but not all Lisp software is free and open source, and even software within Quicklisp has tons of implied version dependencies. Regards, Robert
On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad@symbo1ics.com> wrote:
On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <alessiostalla@gmail.com> wrote:
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Systems are what should get versioned but systems are not what are referred to by source files. Systems largely coordinate the compilation and loading of files. Source files generally have no notion of a system, actually, which may be what is ultimately problematic. Packages are referred to by source files, however. When I said packages, I meant it, and it was not a conflation with the notion of a system.
Why should source files know about systems? Why should packages know about systems?
At the companies I have worked in, we rake in 100s of systems, which consist of n*100s of systems where n is the average number of packages per system. These version collisions happen enough, and we are lucky when they're not silent.
Was it in CL? Was it all in the same process? CL is not designed for these kinds of things.
Often enough, it's an extremely subtle issue that system A depends on {system B at time T1} and system C depends on {system B at time T2}, where T1 and T2 correspond to different versions of system B, even if not explicitly labeled as such.
A and C are incompatible. You need to fix A, B and/or C. The build system can record this incompatibility and issue an early error when you try to build them together. It can't fix A, B and/or C for you.
Let's suppose we simplify the problem however and assume that B has been properly versioned in time, and systems A and C refer properly to the correct version of system B as a dependency. Even then, they will not be able to be simultaneously loaded in a functionally correct fashion. (They *can* both be loaded, but you'll do some clobbering of state along the way.)
No. The build system can error out and tell you the easy way that you have to fix your bugs before you discover it the hard way.
Whether we think about modules as systems or as packages or as whatever, the problem would need to be addressed at the package level, if we are not assuming that we can't synchronize the universe of people writing Common Lisp.
The packages have nothing to do with it, unless you fork B into B1 and B2 (or B1 and B, or B and B2).
Overall, I'd summarize as this. Whether we have the technology in the Common Lisp ecosystem to accommodate this problem or not, it's my opinion and the opinion of many of my colleagues that there is a problem for development teams making extensive use of open and closed source code. There definitely is not a discipline that you see with other languages and their ecosystems (JS/npm, Clojure, etc.) about proper versioning of systems and furthermore proper loading of the proper versions. I don't think that's debatable. The closest we have to this solution is relying on the global nature of a particular Quicklisp distribution, but not all Lisp software is free and open source, and even software within Quicklisp has tons of implied version dependencies.
This is not a very Lisp-specific problem. This is a problem with building software in general. The correct approach is to fix incompatibilities, not to try to deny their existence only to find out the hard way you can't. Google Bazel provides a good way to deterministically build software that follows the correct discipline. So does NixOS. There may be other such tools. Use them. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Toleration is not about respecting other people's ideas. We have every right to fight ideas we think are stupid. Toleration is about respecting other people's persons. We have every duty to respect even persons we think are stupid.
I really honestly felt that this rebuttal was very unhelpful. I basically got from it: - Lisp isn't designed for this kind of use. - Since Lisp wasn't designed for this, the problem is not Lisp, it's your use of Lisp. - To solve the problem (which is not Lisp's), fix (third party) inconsistencies yourself. Because it's not a Lisp problem, two systems depending on two Lisp-simultaneous-incompatible versions of a system is a problem. (Even though each system+dependency combination is perfectly valid in isolation.) - Or, in the end, change your entire operating system to a completely different one with completely different philosophies. This kind of response is just very practically a non-starter for most people. Maybe that means Lisp isn't the right tool for much of the world, like startups, then. Which advances my point, I think, in the original Quora post. (But I admit this thread is about package or system versioning, not the original Quora point.) I wrote some stuff inline below. On Wed, May 18, 2016 at 10:14 PM, Faré <fahree@gmail.com> wrote:
On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad@symbo1ics.com> wrote:
On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <alessiostalla@gmail.com
wrote:
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages"
in
the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Systems are what should get versioned but systems are not what are referred to by source files. Systems largely coordinate the compilation and loading of files. Source files generally have no notion of a system, actually, which may be what is ultimately problematic. Packages are referred to by source files, however. When I said packages, I meant it, and it was not a conflation with the notion of a system.
Why should source files know about systems? Why should packages know about systems?
Maybe they should, maybe they shouldn't. But the crappy packages-as-namespaces system isn't cutting it for large programs, and modules usually naturally have an associated namespace, but there's no formal correspondence between packages and systems. But more often than not there is an implicit association, judging by how most modern Common Lisp libraries are written. So maybe there's value in taking advantage of that correspondence.
At the companies I have worked in, we rake in 100s of systems, which consist of n*100s of systems where n is the average number of packages per system. These version collisions happen enough, and we are lucky when they're not silent.
Was it in CL? Was it all in the same process? CL is not designed for these kinds of things.
CL in the same process. CL very clearly wasn't designed for a time where 100s of packages would be available and simultaneously usable. But I'm not ready to concede and say "well, Lisp is only good for five or six massive packages or systems written by a few guys, like the good ol' days".
Often enough, it's an extremely subtle issue that system A depends on {system B at time T1} and system C depends on {system B at time T2}, where T1 and T2 correspond to different versions of system B, even if not explicitly labeled as such.
A and C are incompatible. You need to fix A, B and/or C. The build system can record this incompatibility and issue an early error when you try to build them together. It can't fix A, B and/or C for you.
A and C are incompatible only because of this whole thing we are talking about. Obviously they're incompatible. That's the whole problem. Saying "Fix A, B, C" is, to me, declaring that this whole thing isn't a problem, and the problem is library writers. Why can't A and C depend on whatever they want at some time? They of course can, but it's non-sense to think people will rename their namespaces every iteration of their software. That's just not how Lisp code has been written.
Let's suppose we simplify the problem however and assume that B has been properly versioned in time, and systems A and C refer properly to the correct version of system B as a dependency. Even then, they will not be able to be simultaneously loaded in a functionally correct fashion. (They *can* both be loaded, but you'll do some clobbering of state along the way.)
No. The build system can error out and tell you the easy way that you have to fix your bugs before you discover it the hard way
The build system erroring would be a step above the status quo. But maybe there's not actually a logical error. We are only considering it an error because of the way packages work in Lisp, no?
Whether we think about modules as systems or as packages or as whatever, the
problem would need to be addressed at the package level, if we are not assuming that we can't synchronize the universe of people writing Common Lisp.
The packages have nothing to do with it, unless you fork B into B1 and B2 (or B1 and B, or B and B2).
Overall, I'd summarize as this. Whether we have the technology in the Common Lisp ecosystem to accommodate this problem or not, it's my opinion and the opinion of many of my colleagues that there is a problem for development teams making extensive use of open and closed source code. There definitely is not a discipline that you see with other languages and their ecosystems (JS/npm, Clojure, etc.) about proper versioning of systems and furthermore proper loading of the proper versions. I don't think that's debatable. The closest we have to this solution is relying on the global nature of a particular Quicklisp distribution, but not all Lisp software is free and open source, and even software within Quicklisp has tons of implied version dependencies.
This is not a very Lisp-specific problem. This is a problem with building software in general. The correct approach is to fix incompatibilities, not to try to deny their existence only to find out the hard way you can't.
Google Bazel provides a good way to deterministically build software that follows the correct discipline. So does NixOS. There may be other such tools. Use them.
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems. I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool. I don't know. The original question I was answering was why Lisp isn't what we are all using today, and I think this weird thread serves as additional evidence. Am I way off base here? Cheers, Robert
I don't understand when you say this is "not an issue that one runs into writing in another language and associated ecosystem". AFAIK, in Java, you can have only one version of a given package loaded into the JVM at any one time. Am I missing something here? -- Scott On Wed, May 18, 2016 at 11:41 PM, Robert Smith <quad@symbo1ics.com> wrote:
I really honestly felt that this rebuttal was very unhelpful. I basically got from it:
- Lisp isn't designed for this kind of use. - Since Lisp wasn't designed for this, the problem is not Lisp, it's your use of Lisp. - To solve the problem (which is not Lisp's), fix (third party) inconsistencies yourself. Because it's not a Lisp problem, two systems depending on two Lisp-simultaneous-incompatible versions of a system is a problem. (Even though each system+dependency combination is perfectly valid in isolation.) - Or, in the end, change your entire operating system to a completely different one with completely different philosophies.
This kind of response is just very practically a non-starter for most people. Maybe that means Lisp isn't the right tool for much of the world, like startups, then. Which advances my point, I think, in the original Quora post. (But I admit this thread is about package or system versioning, not the original Quora point.)
I wrote some stuff inline below.
On Wed, May 18, 2016 at 10:14 PM, Faré <fahree@gmail.com> wrote:
On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad@symbo1ics.com> wrote:
On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <
wrote:
The stance on packages found in the mentioned Quora post is based on
alessiostalla@gmail.com> the
old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Systems are what should get versioned but systems are not what are referred to by source files. Systems largely coordinate the compilation and loading of files. Source files generally have no notion of a system, actually, which may be what is ultimately problematic. Packages are referred to by source files, however. When I said packages, I meant it, and it was not a conflation with the notion of a system.
Why should source files know about systems? Why should packages know about systems?
Maybe they should, maybe they shouldn't. But the crappy packages-as-namespaces system isn't cutting it for large programs, and modules usually naturally have an associated namespace, but there's no formal correspondence between packages and systems. But more often than not there is an implicit association, judging by how most modern Common Lisp libraries are written. So maybe there's value in taking advantage of that correspondence.
At the companies I have worked in, we rake in 100s of systems, which consist of n*100s of systems where n is the average number of packages per system. These version collisions happen enough, and we are lucky when they're not silent.
Was it in CL? Was it all in the same process? CL is not designed for these kinds of things.
CL in the same process.
CL very clearly wasn't designed for a time where 100s of packages would be available and simultaneously usable.
But I'm not ready to concede and say "well, Lisp is only good for five or six massive packages or systems written by a few guys, like the good ol' days".
Often enough, it's an extremely subtle issue that system A depends on {system B at time T1} and system C depends on {system B at time T2}, where T1 and T2 correspond to different versions of system B, even if not explicitly labeled as such.
A and C are incompatible. You need to fix A, B and/or C. The build system can record this incompatibility and issue an early error when you try to build them together. It can't fix A, B and/or C for you.
A and C are incompatible only because of this whole thing we are talking about. Obviously they're incompatible. That's the whole problem. Saying "Fix A, B, C" is, to me, declaring that this whole thing isn't a problem, and the problem is library writers. Why can't A and C depend on whatever they want at some time? They of course can, but it's non-sense to think people will rename their namespaces every iteration of their software. That's just not how Lisp code has been written.
Let's suppose we simplify the problem however and assume that B has been properly versioned in time, and systems A and C refer properly to the correct version of system B as a dependency. Even then, they will not be able to be simultaneously loaded in a functionally correct fashion. (They *can* both be loaded, but you'll do some clobbering of state along the way.)
No. The build system can error out and tell you the easy way that you have to fix your bugs before you discover it the hard way
The build system erroring would be a step above the status quo. But maybe there's not actually a logical error. We are only considering it an error because of the way packages work in Lisp, no?
Whether we think about modules as systems or as packages or as whatever, the
problem would need to be addressed at the package level, if we are not assuming that we can't synchronize the universe of people writing Common Lisp.
The packages have nothing to do with it, unless you fork B into B1 and B2 (or B1 and B, or B and B2).
Overall, I'd summarize as this. Whether we have the technology in the Common Lisp ecosystem to accommodate this problem or not, it's my opinion and the opinion of many of my colleagues that there is a problem for development teams making extensive use of open and closed source code. There definitely is not a discipline that you see with other languages and their ecosystems (JS/npm, Clojure, etc.) about proper versioning of systems and furthermore proper loading of the proper versions. I don't think that's debatable. The closest we have to this solution is relying on the global nature of a particular Quicklisp distribution, but not all Lisp software is free and open source, and even software within Quicklisp has tons of implied version dependencies.
This is not a very Lisp-specific problem. This is a problem with building software in general. The correct approach is to fix incompatibilities, not to try to deny their existence only to find out the hard way you can't.
Google Bazel provides a good way to deterministically build software that follows the correct discipline. So does NixOS. There may be other such tools. Use them.
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
I don't know. The original question I was answering was why Lisp isn't what we are all using today, and I think this weird thread serves as additional evidence.
Am I way off base here?
Cheers,
Robert
I don't mean any other language, but I do mean any other relatively modern one. Rust, JavaScript, Clojure are maybe some examples off the top of my head. Either modules are somehow first class (by way of having modules-as-function-dictionaries), or the respective packaging/loading/whatever systems will at least let you know that about unsatisfied/mismatched versions. I think in light of this discussion you can boil down one's opinion on this matter to one of three alternatives: 1. Lisp+ecosystem and associated libraries are problematic in that it's not possible to load multiple versions of a library due to the way CL packages work. (Packages or systems aren't first-class modules. Namespace names are global and aren't late-bound.) 2. There's something like a SocialProblem (TM) where Lispers lack (a) versioning discipline, (b) dependency versioning discipline which exacerbates issues like inadvertent multiple loading, or incompatible loading of libraries. 3. There's actually no issue in principle: inconsistencies need to "just" get fixed by the library vendor/user, or better tooling needs to be used to track your project's direct and transitive dependencies. It's my personal opinion, as is evident in previous messages, that I think 1 & 2 are issues. Some additional food for thought: * Lisp is old. Some libraries go back to the 80s. * Not everything is very strictly maintained. Some useful libraries haven't been updated for some 5-10 years. * Like many other languages, Java is continuously updated, and as are the libraries of its large standard set. While Java/etc libraries certainly can and do depend on third party libraries, the majority of depended-on libraries are standard ones. This is obviously not the case for Lisp. * Because library distribution wasn't a relatively solved problem until recently in Lisp history, some library authors unfortunately include copies of Lisp libraries as a part of their library tree. * Hot patching libraries makes this all a ton more complicated in principle. * Visible libraries are global to the entire Lisp universe. (Unlike Python where you can, just within a file, import and late-bind a namespace.) * Tangential: the whole name/nickname collision issue. Package-local nicknames are a language extension which could solve this, even if not (yet?) universally used. Making it easier to work in this large, old, and diverse ecosystem of implementations and libraries seems important to me, and versioning is one useful and tangible way to make things easier. Robert On Wednesday, May 18, 2016, Scott L. Burson <Scott@sympoiesis.com> wrote:
I don't understand when you say this is "not an issue that one runs into writing in another language and associated ecosystem". AFAIK, in Java, you can have only one version of a given package loaded into the JVM at any one time. Am I missing something here?
-- Scott
On Wed, May 18, 2016 at 11:41 PM, Robert Smith <quad@symbo1ics.com <javascript:_e(%7B%7D,'cvml','quad@symbo1ics.com');>> wrote:
I really honestly felt that this rebuttal was very unhelpful. I basically got from it:
- Lisp isn't designed for this kind of use. - Since Lisp wasn't designed for this, the problem is not Lisp, it's your use of Lisp. - To solve the problem (which is not Lisp's), fix (third party) inconsistencies yourself. Because it's not a Lisp problem, two systems depending on two Lisp-simultaneous-incompatible versions of a system is a problem. (Even though each system+dependency combination is perfectly valid in isolation.) - Or, in the end, change your entire operating system to a completely different one with completely different philosophies.
This kind of response is just very practically a non-starter for most people. Maybe that means Lisp isn't the right tool for much of the world, like startups, then. Which advances my point, I think, in the original Quora post. (But I admit this thread is about package or system versioning, not the original Quora point.)
I wrote some stuff inline below.
On Wed, May 18, 2016 at 10:14 PM, Faré <fahree@gmail.com <javascript:_e(%7B%7D,'cvml','fahree@gmail.com');>> wrote:
On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad@symbo1ics.com <javascript:_e(%7B%7D,'cvml','quad@symbo1ics.com');>> wrote:
On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <
wrote:
The stance on packages found in the mentioned Quora post is based on
old misconception about packages being modules, or "software
the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that
alessiostalla@gmail.com <javascript:_e(%7B%7D,'cvml','alessiostalla@gmail.com');>> the packages" in they have
versions. Software has versions, not names.
Systems are what should get versioned but systems are not what are referred to by source files. Systems largely coordinate the compilation and loading of files. Source files generally have no notion of a system, actually, which may be what is ultimately problematic. Packages are referred to by source files, however. When I said packages, I meant it, and it was not a conflation with the notion of a system.
Why should source files know about systems? Why should packages know about systems?
Maybe they should, maybe they shouldn't. But the crappy packages-as-namespaces system isn't cutting it for large programs, and modules usually naturally have an associated namespace, but there's no formal correspondence between packages and systems. But more often than not there is an implicit association, judging by how most modern Common Lisp libraries are written. So maybe there's value in taking advantage of that correspondence.
At the companies I have worked in, we rake in 100s of systems, which consist of n*100s of systems where n is the average number of packages per system. These version collisions happen enough, and we are lucky when they're not silent.
Was it in CL? Was it all in the same process? CL is not designed for these kinds of things.
CL in the same process.
CL very clearly wasn't designed for a time where 100s of packages would be available and simultaneously usable.
But I'm not ready to concede and say "well, Lisp is only good for five or six massive packages or systems written by a few guys, like the good ol' days".
Often enough, it's an extremely subtle issue that system A depends on {system B at time T1} and system C depends on {system B at time T2}, where T1 and T2 correspond to different versions of system B, even if not explicitly labeled as such.
A and C are incompatible. You need to fix A, B and/or C. The build system can record this incompatibility and issue an early error when you try to build them together. It can't fix A, B and/or C for you.
A and C are incompatible only because of this whole thing we are talking about. Obviously they're incompatible. That's the whole problem. Saying "Fix A, B, C" is, to me, declaring that this whole thing isn't a problem, and the problem is library writers. Why can't A and C depend on whatever they want at some time? They of course can, but it's non-sense to think people will rename their namespaces every iteration of their software. That's just not how Lisp code has been written.
Let's suppose we simplify the problem however and assume that B has been properly versioned in time, and systems A and C refer properly to the correct version of system B as a dependency. Even then, they will not be able to be simultaneously loaded in a functionally correct fashion. (They *can* both be loaded, but you'll do some clobbering of state along the way.)
No. The build system can error out and tell you the easy way that you have to fix your bugs before you discover it the hard way
The build system erroring would be a step above the status quo. But maybe there's not actually a logical error. We are only considering it an error because of the way packages work in Lisp, no?
Whether we think about modules as systems or as packages or as whatever, the
problem would need to be addressed at the package level, if we are not assuming that we can't synchronize the universe of people writing Common Lisp.
The packages have nothing to do with it, unless you fork B into B1 and B2 (or B1 and B, or B and B2).
Overall, I'd summarize as this. Whether we have the technology in the Common Lisp ecosystem to accommodate this problem or not, it's my opinion and the opinion of many of my colleagues that there is a problem for development teams making extensive use of open and closed source code. There definitely is not a discipline that you see with other languages and their ecosystems (JS/npm, Clojure, etc.) about proper versioning of systems and furthermore proper loading of the proper versions. I don't think that's debatable. The closest we have to this solution is relying on the global nature of a particular Quicklisp distribution, but not all Lisp software is free and open source, and even software within Quicklisp has tons of implied version dependencies.
This is not a very Lisp-specific problem. This is a problem with building software in general. The correct approach is to fix incompatibilities, not to try to deny their existence only to find out the hard way you can't.
Google Bazel provides a good way to deterministically build software that follows the correct discipline. So does NixOS. There may be other such tools. Use them.
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
I don't know. The original question I was answering was why Lisp isn't what we are all using today, and I think this weird thread serves as additional evidence.
Am I way off base here?
Cheers,
Robert
I don't mean any other language, but I do mean any other relatively modern one. Rust, JavaScript, Clojure are maybe some examples off the top of my head.
Hans gave a good example about Clojure. Javascript, with Node.js/npm, is in an even worse position.
Either modules are somehow first class (by way of having modules-as-function- dictionaries), or the respective packaging/loading/whatever systems will at least let you know that about unsatisfied/mismatched versions.
I think in light of this discussion you can boil down one's opinion on this matter to one of three alternatives:
1. Lisp+ecosystem and associated libraries are problematic in that it's not possible to load multiple versions of a library due to the way CL packages work. (Packages or systems aren't first-class modules. Namespace names are global and aren't late-bound.)
Hans described the QA problem with loading two versions of the "same" library in the same process: if data structures from one version are fed into the other version, all sorts of problems can happen, and worse, they can happen without noticing. Think of what can happen if the semantics of a string or integer slot changes.
2. There's something like a SocialProblem (TM) where Lispers lack (a) versioning discipline, (b) dependency versioning discipline which exacerbates issues like inadvertent multiple loading, or incompatible loading of libraries.
Semantic versioning won't help you without automatic tools to determine if the API of a library changed in an incompatible way, e.g. if a function removed a keyword arg, or changed the type of an existing arg.
3. There's actually no issue in principle: inconsistencies need to "just" get fixed by the library vendor/user, or better tooling needs to be used to track your project's direct and transitive dependencies.
It's my personal opinion, as is evident in previous messages, that I think 1 & 2 are issues.
You didn't give an example of how exactly those two things are a problem, and how adding those features would solve the problem(s).
Some additional food for thought:
* Lisp is old. Some libraries go back to the 80s.
* Not everything is very strictly maintained. Some useful libraries haven't been updated for some 5-10 years.
What useful libraries that actually need updates haven't been updated ?
* Like many other languages, Java is continuously updated, and as are the libraries of its large standard set. While Java/etc libraries certainly can and do depend on third party libraries, the majority of depended-on libraries are standard ones. This is obviously not the case for Lisp.
Just look into Maven packages. Large parts of the Java standard library are obsolete and people use third party libraries(just like with CL). In some cases those libraries make it back into the standard (http://www.joda.org/joda-time/), but that's rare.
* Because library distribution wasn't a relatively solved problem until recently in Lisp history, some library authors unfortunately include copies of Lisp libraries as a part of their library tree.
That's very rare, in my experience. Do you have examples ?
* Hot patching libraries makes this all a ton more complicated in principle.
Where did you see this?
* Visible libraries are global to the entire Lisp universe. (Unlike Python where you can, just within a file, import and late-bind a namespace.)
* Tangential: the whole name/nickname collision issue. Package-local nicknames are a language extension which could solve this, even if not (yet?) universally used.
Local nicknames would make certain things more convenient, but what does that have to do with the original question, versioning ?
Making it easier to work in this large, old, and diverse ecosystem of implementations and libraries seems important to me, and versioning is one useful and tangible way to make things easier.
Robert
On Wednesday, May 18, 2016, Scott L. Burson <Scott@sympoiesis.com> wrote:
I don't understand when you say this is "not an issue that one runs into writing in another language and associated ecosystem". AFAIK, in Java, you can have only one version of a given package loaded into the JVM at any one time. Am I missing something here? -- Scott
On Wed, May 18, 2016 at 11:41 PM, Robert Smith <quad@symbo1ics.com> wrote:
I really honestly felt that this rebuttal was very unhelpful. I basically got from it: * Lisp isn't designed for this kind of use. * Since Lisp wasn't designed for this, the problem is not Lisp, it's your use of Lisp. * To solve the problem (which is not Lisp's), fix (third party) inconsistencies yourself. Because it's not a Lisp problem, two systems depending on two Lisp-simultaneous-incompatible versions of a system is a problem. (Even though each system+dependency combination is perfectly valid in isolation.) * Or, in the end, change your entire operating system to a completely different one with completely different philosophies. This kind of response is just very practically a non-starter for most people. Maybe that means Lisp isn't the right tool for much of the world, like startups, then. Which advances my point, I think, in the original Quora post. (But I admit this thread is about package or system versioning, not the original Quora point.)
I wrote some stuff inline below.
On Wed, May 18, 2016 at 10:14 PM, Faré <fahree@gmail.com> wrote:
On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad@symbo1ics.com> wrote:
On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <alessiostalla@gmail.com> wrote:
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages" in the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Systems are what should get versioned but systems are not what are referred to by source files. Systems largely coordinate the compilation and loading of files. Source files generally have no notion of a system, actually, which may be what is ultimately problematic. Packages are referred to by source files, however. When I said packages, I meant it, and it was not a conflation with the notion of a system.
Why should source files know about systems? Why should packages know about systems?
Maybe they should, maybe they shouldn't. But the crappy packages-as- namespaces system isn't cutting it for large programs, and modules usually naturally have an associated namespace, but there's no formal correspondence between packages and systems. But more often than not there is an implicit association, judging by how most modern Common Lisp libraries are written. So maybe there's value in taking advantage of that correspondence.
At the companies I have worked in, we rake in 100s of systems, which consist of n*100s of systems where n is the average number of packages per system. These version collisions happen enough, and we are lucky when they're not silent.
Was it in CL? Was it all in the same process? CL is not designed for these kinds of things.
CL in the same process.
CL very clearly wasn't designed for a time where 100s of packages would be available and simultaneously usable.
But I'm not ready to concede and say "well, Lisp is only good for five or six massive packages or systems written by a few guys, like the good ol' days".
Often enough, it's an extremely subtle issue that system A depends on {system B at time T1} and system C depends on {system B at time T2}, where T1 and T2 correspond to different versions of system B, even if not explicitly labeled as such.
A and C are incompatible. You need to fix A, B and/or C. The build system can record this incompatibility and issue an early error when you try to build them together. It can't fix A, B and/or C for you.
A and C are incompatible only because of this whole thing we are talking about. Obviously they're incompatible. That's the whole problem. Saying "Fix A, B, C" is, to me, declaring that this whole thing isn't a problem, and the problem is library writers. Why can't A and C depend on whatever they want at some time? They of course can, but it's non-sense to think people will rename their namespaces every iteration of their software. That's just not how Lisp code has been written.
Let's suppose we simplify the problem however and assume that B has been properly versioned in time, and systems A and C refer properly to the correct version of system B as a dependency. Even then, they will not be able to be simultaneously loaded in a functionally correct fashion. (They *can* both be loaded, but you'll do some clobbering of state along the way.)
No. The build system can error out and tell you the easy way that you have to fix your bugs before you discover it the hard way
The build system erroring would be a step above the status quo. But maybe there's not actually a logical error. We are only considering it an error because of the way packages work in Lisp, no?
Whether we think about modules as systems or as packages or as whatever, the problem would need to be addressed at the package level, if we are not assuming that we can't synchronize the universe of people writing Common Lisp.
The packages have nothing to do with it, unless you fork B into B1 and B2 (or B1 and B, or B and B2).
Overall, I'd summarize as this. Whether we have the technology in the Common Lisp ecosystem to accommodate this problem or not, it's my opinion and the opinion of many of my colleagues that there is a problem for development teams making extensive use of open and closed source code. There definitely is not a discipline that you see with other languages and their ecosystems (JS/npm, Clojure, etc.) about proper versioning of systems and furthermore proper loading of the proper versions. I don't think that's debatable. The closest we have to this solution is relying on the global nature of a particular Quicklisp distribution, but not all Lisp software is free and open source, and even software within Quicklisp has tons of implied version dependencies.
This is not a very Lisp-specific problem. This is a problem with building software in general. The correct approach is to fix incompatibilities, not to try to deny their existence only to find out the hard way you can't.
Google Bazel provides a good way to deterministically build software that follows the correct discipline. So does NixOS. There may be other such tools. Use them.
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
I don't know. The original question I was answering was why Lisp isn't what we are all using today, and I think this weird thread serves as additional evidence.
Am I way off base here?
Cheers,
Robert
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Inline clarifications. On Thursday, May 19, 2016, Stelian Ionescu <sionescu@cddr.org> wrote:
I don't mean any other language, but I do mean any other relatively modern one. Rust, JavaScript, Clojure are maybe some examples off the top of my head.
Hans gave a good example about Clojure. Javascript, with Node.js/npm, is in an even worse position.
Either modules are somehow first class (by way of having modules-as-function-dictionaries), or the respective packaging/loading/whatever systems will at least let you know that about unsatisfied/mismatched versions.
I think in light of this discussion you can boil down one's opinion on this matter to one of three alternatives:
1. Lisp+ecosystem and associated libraries are problematic in that it's not possible to load multiple versions of a library due to the way CL packages work. (Packages or systems aren't first-class modules. Namespace names are global and aren't late-bound.)
Hans described the QA problem with loading two versions of the "same" library in the same process: if data structures from one version are fed into the other version, all sorts of problems can happen, and worse, they can happen without noticing. Think of what can happen if the semantics of a string or integer slot changes.
See below.
2. There's something like a SocialProblem (TM) where Lispers lack (a) versioning discipline, (b) dependency versioning discipline which exacerbates issues like inadvertent multiple loading, or incompatible loading of libraries.
Semantic versioning won't help you without automatic tools to determine if the API of a library changed in an incompatible way, e.g. if a function removed a keyword arg, or changed the type of an existing arg.
I would say that semvers should in principle help you detect incompatibilities, but as you hinted with "illusion", maybe these in practice can't be trusted or relied upon generally.
3. There's actually no issue in principle: inconsistencies need to "just" get fixed by the library vendor/user, or better tooling needs to be used to track your project's direct and transitive dependencies.
It's my personal opinion, as is evident in previous messages, that I think 1 & 2 are issues.
You didn't give an example of how exactly those two things are a problem, and how adding those features would solve the problem(s).
#1: Assuming that there isn't incompatible passing of data, if A depends on B1 and C depends on B2, and in isolation A-B1 works/passes tests, and C-B2 works/passes tests, then A-B1 and C-B2 loaded together (assuming a non-existent Lisp) should work fine. The problem being addressed is lack of synchrony in the universe of Lisp modules. (My first-order thought is that even incompatible data wouldn't be an issue if those data types are properly encapsulated in classes or structures, by virtue of the fact that those types become distinct by way of different names. I envision that only untyped things could cause problems, like [ap]-lists, strings, etc.) #2: If there was diligence is properly versioning libraries, and diligence in specifying dependency version requirements, then detection of issues could happen. This is solving recognition of a possible issue.
Some additional food for thought:
* Lisp is old. Some libraries go back to the 80s.
* Not everything is very strictly maintained. Some useful libraries haven't been updated for some 5-10 years.
What useful libraries that actually need updates haven't been updated ?
I'll get back to this when I'm at my computer next.
* Like many other languages, Java is continuously updated, and as are the libraries of its large standard set. While Java/etc libraries certainly can and do depend on third party libraries, the majority of depended-on libraries are standard ones. This is obviously not the case for Lisp.
Just look into Maven packages. Large parts of the Java standard library are obsolete and people use third party libraries(just like with CL). In some cases those libraries make it back into the standard( http://www.joda.org/joda-time/), but that's rare.
* Because library distribution wasn't a relatively solved problem until recently in Lisp history, some library authors unfortunately include copies of Lisp libraries as a part of their library tree.
That's very rare, in my experience. Do you have examples ?
* Hot patching libraries makes this all a ton more complicated in principle.
Where did you see this?
For these last two questions, I don't have the examples off hand and unfortunately it was at a company I no longer work at. Including: ALEXANDRIA has been included directly before. I don't remember which library it was. I only detected it because of function redefinition warnings on first uncached build. There were other examples but I don't recall. Hot Patching: I think (??) some linear algebra library was being hot patched for some specific use of that library. * Visible libraries are global to the entire Lisp universe. (Unlike Python
where you can, just within a file, import and late-bind a namespace.)
* Tangential: the whole name/nickname collision issue. Package-local nicknames are a language extension which could solve this, even if not (yet?) universally used.
Local nicknames would make certain things more convenient, but what does that have to do with the original question, versioning ?
It was tangential to packages and names of things.
Making it easier to work in this large, old, and diverse ecosystem of implementations and libraries seems important to me, and versioning is one useful and tangible way to make things easier.
Robert
On Wednesday, May 18, 2016, Scott L. Burson <Scott@sympoiesis.com <javascript:_e(%7B%7D,'cvml','Scott@sympoiesis.com');>> wrote:
I don't understand when you say this is "not an issue that one runs into writing in another language and associated ecosystem". AFAIK, in Java, you can have only one version of a given package loaded into the JVM at any one time. Am I missing something here? -- Scott
On Wed, May 18, 2016 at 11:41 PM, Robert Smith <quad@symbo1ics.com> wrote:
I really honestly felt that this rebuttal was very unhelpful. I basically got from it:
- Lisp isn't designed for this kind of use. - Since Lisp wasn't designed for this, the problem is not Lisp, it's your use of Lisp. - To solve the problem (which is not Lisp's), fix (third party) inconsistencies yourself. Because it's not a Lisp problem, two systems depending on two Lisp-simultaneous-incompatible versions of a system is a problem. (Even though each system+dependency combination is perfectly valid in isolation.) - Or, in the end, change your entire operating system to a completely different one with completely different philosophies.
This kind of response is just very practically a non-starter for most people. Maybe that means Lisp isn't the right tool for much of the world, like startups, then. Which advances my point, I think, in the original Quora post. (But I admit this thread is about package or system versioning, not the original Quora point.)
I wrote some stuff inline below.
On Wed, May 18, 2016 at 10:14 PM, Faré <fahree@gmail.com> wrote:
On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad@symbo1ics.com> wrote:
On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla <alessiostalla@gmail.com
wrote:
The stance on packages found in the mentioned Quora post is based on the old misconception about packages being modules, or "software packages"
in
the Linux distribution sense. They're not. They are really just namespaces, containers of symbols. So it does not make any sense, to me, that they have versions. Software has versions, not names.
Systems are what should get versioned but systems are not what are referred to by source files. Systems largely coordinate the compilation and loading of files. Source files generally have no notion of a system, actually, which may be what is ultimately problematic. Packages are referred to by source files, however. When I said packages, I meant it, and it was not a conflation with the notion of a system.
Why should source files know about systems? Why should packages know about systems?
Maybe they should, maybe they shouldn't. But the crappy packages-as-namespaces system isn't cutting it for large programs, and modules usually naturally have an associated namespace, but there's no formal correspondence between packages and systems. But more often than not there is an implicit association, judging by how most modern Common Lisp libraries are written. So maybe there's value in taking advantage of that correspondence.
At the companies I have worked in, we rake in 100s of systems, which consist of n*100s of systems where n is the average number of packages per system. These version collisions happen enough, and we are lucky when they're not silent.
Was it in CL? Was it all in the same process? CL is not designed for these kinds of things.
CL in the same process.
CL very clearly wasn't designed for a time where 100s of packages would be available and simultaneously usable.
But I'm not ready to concede and say "well, Lisp is only good for five or six massive packages or systems written by a few guys, like the good ol' days".
Often enough, it's an extremely subtle issue that system A depends on {system B at time T1} and system C depends on {system B at time T2}, where T1 and T2 correspond to different versions of system B, even if not explicitly labeled as such.
A and C are incompatible. You need to fix A, B and/or C. The build system can record this incompatibility and issue an early error when you try to build them together. It can't fix A, B and/or C for you.
A and C are incompatible only because of this whole thing we are talking about. Obviously they're incompatible. That's the whole problem. Saying "Fix A, B, C" is, to me, declaring that this whole thing isn't a problem, and the problem is library writers. Why can't A and C depend on whatever they want at some time? They of course can, but it's non-sense to think people will rename their namespaces every iteration of their software. That's just not how Lisp code has been written.
Let's suppose we simplify the problem however and assume that B has been properly versioned in time, and systems A and C refer properly to the correct version of system B as a dependency. Even then, they will not be able to be simultaneously loaded in a functionally correct fashion. (They *can* both be loaded, but you'll do some clobbering of state along the way.)
No. The build system can error out and tell you the easy way that you have to fix your bugs before you discover it the hard way
The build system erroring would be a step above the status quo. But maybe there's not actually a logical error. We are only considering it an error because of the way packages work in Lisp, no?
Whether we think about modules as systems or as packages or as whatever, the problem would need to be addressed at the package level, if we are not assuming that we can't synchronize the universe of people writing Common Lisp.
The packages have nothing to do with it, unless you fork B into B1 and B2 (or B1 and B, or B and B2).
Overall, I'd summarize as this. Whether we have the technology in the Common Lisp ecosystem to accommodate this problem or not, it's my opinion and the opinion of many of my colleagues that there is a problem for development teams making extensive use of open and closed source code. There definitely is not a discipline that you see with other languages and their ecosystems (JS/npm, Clojure, etc.) about proper versioning of systems and furthermore proper loading of the proper versions. I don't think that's debatable. The closest we have to this solution is relying on the global nature of a particular Quicklisp distribution, but not all Lisp software is free and open source, and even software within Quicklisp has tons of implied version dependencies.
This is not a very Lisp-specific problem. This is a problem with building software in general. The correct approach is to fix incompatibilities, not to try to deny their existence only to find out the hard way you can't.
Google Bazel provides a good way to deterministically build software that follows the correct discipline. So does NixOS. There may be other such tools. Use them.
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
I don't know. The original question I was answering was why Lisp isn't what we are all using today, and I think this weird thread serves as additional evidence.
Am I way off base here?
Cheers,
Robert
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Inline clarifications.
On Thursday, May 19, 2016, Stelian Ionescu <sionescu@cddr.org> wrote:
__
I don't mean any other language, but I do mean any other relatively modern one. Rust, JavaScript, Clojure are maybe some examples off the top of my head.
Hans gave a good example about Clojure. Javascript, with Node.js/npm, is in an even worse position.
Either modules are somehow first class (by way of having modules-as-function- dictionaries), or the respective packaging/loading/whatever systems will at least let you know that about unsatisfied/mismatched versions.
I think in light of this discussion you can boil down one's opinion on this matter to one of three alternatives:
1. Lisp+ecosystem and associated libraries are problematic in that it's not possible to load multiple versions of a library due to the way CL packages work. (Packages or systems aren't first-class modules. Namespace names are global and aren't late-bound.)
Hans described the QA problem with loading two versions of the "same" library in the same process: if data structures from one version are fed into the other version, all sorts of problems can happen, and worse, they can happen without noticing. Think of what can happen if the semantics of a string or integer slot changes.
See below.
2. There's something like a SocialProblem (TM) where Lispers lack (a) versioning discipline, (b) dependency versioning discipline which exacerbates issues like inadvertent multiple loading, or incompatible loading of libraries.
Semantic versioning won't help you without automatic tools to determine if the API of a library changed in an incompatible way, e.g. if a function removed a keyword arg, or changed the type of an existing arg.
I would say that semvers should in principle help you detect incompatibilities, but as you hinted with "illusion", maybe these in practice can't be trusted or relied upon generally.
3. There's actually no issue in principle: inconsistencies need to "just" get fixed by the library vendor/user, or better tooling needs to be used to track your project's direct and transitive dependencies.
It's my personal opinion, as is evident in previous messages, that I think 1 & 2 are issues.
You didn't give an example of how exactly those two things are a problem, and how adding those features would solve the problem(s).
#1: Assuming that there isn't incompatible passing of data, if A # depends on B1 and C depends on B2, and in isolation A-B1 # works/passes tests, and C-B2 works/passes tests, then A-B1 and C- # B2 loaded together (assuming a non-existent Lisp) should work # fine. The problem being addressed is lack of synchrony in the # universe of Lisp modules.
(My first-order thought is that even incompatible data wouldn't be an issue if those data types are properly encapsulated in classes or structures, by virtue of the fact that those types become distinct by way of different names. I envision that only untyped things could cause problems, like [ap]-lists, strings, etc.)
#2: If there was diligence is properly versioning libraries, and # diligence in specifying dependency version requirements, then # detection of issues could happen. This is solving recognition of a # possible issue.
Some additional food for thought:
* Lisp is old. Some libraries go back to the 80s.
* Not everything is very strictly maintained. Some useful libraries haven't been updated for some 5-10 years.
What useful libraries that actually need updates haven't been updated ?
I'll get back to this when I'm at my computer next.
* Like many other languages, Java is continuously updated, and as are the libraries of its large standard set. While Java/etc libraries certainly can and do depend on third party libraries, the majority of depended-on libraries are standard ones. This is obviously not the case for Lisp.
Just look into Maven packages. Large parts of the Java standard library are obsolete and people use third party libraries(just like with CL). In some cases those libraries make it back into the standard (http://www.joda.org/joda-time/), but that's rare.
* Because library distribution wasn't a relatively solved problem until recently in Lisp history, some library authors unfortunately include copies of Lisp libraries as a part of their library tree.
That's very rare, in my experience. Do you have examples ?
* Hot patching libraries makes this all a ton more complicated in principle.
Where did you see this?
For these last two questions, I don't have the examples off hand and unfortunately it was at a company I no longer work at.
Including: ALEXANDRIA has been included directly before. I don't remember which library it was. I only detected it because of function redefinition warnings on first uncached build. There were other examples but I don't recall.
Hot Patching: I think (??) some linear algebra library was being hot patched for some specific use of that library.
* Visible libraries are global to the entire Lisp universe. (Unlike Python where you can, just within a file, import and late-bind a namespace.)
* Tangential: the whole name/nickname collision issue. Package-local nicknames are a language extension which could solve this, even if not (yet?) universally used.
Local nicknames would make certain things more convenient, but what does that have to do with the original question, versioning ?
It was tangential to packages and names of things.
Making it easier to work in this large, old, and diverse ecosystem of implementations and libraries seems important to me, and versioning is one useful and tangible way to make things easier.
Robert
On Wednesday, May 18, 2016, Scott L. Burson <Scott@sympoiesis.com> wrote:
I don't understand when you say this is "not an issue that one runs into writing in another language and associated ecosystem". AFAIK, in Java, you can have only one version of a given package loaded into the JVM at any one time. Am I missing something here? -- Scott
On Wed, May 18, 2016 at 11:41 PM, Robert Smith <quad@symbo1ics.com> wrote:
I really honestly felt that this rebuttal was very unhelpful. I basically got from it: * Lisp isn't designed for this kind of use. * Since Lisp wasn't designed for this, the problem is not Lisp, it's your use of Lisp. * To solve the problem (which is not Lisp's), fix (third party) inconsistencies yourself. Because it's not a Lisp problem, two systems depending on two Lisp-simultaneous-incompatible versions of a system is a problem. (Even though each system+dependency combination is perfectly valid in isolation.) * Or, in the end, change your entire operating system to a completely different one with completely different philosophies. This kind of response is just very practically a non-starter for most people. Maybe that means Lisp isn't the right tool for much of the world, like startups, then. Which advances my point, I think, in the original Quora post. (But I admit this thread is about package or system versioning, not the original Quora point.)
I wrote some stuff inline below.
On Wed, May 18, 2016 at 10:14 PM, Faré <fahree@gmail.com> wrote:
On Wed, May 18, 2016 at 5:49 PM, Robert Smith <quad@symbo1ics.com> wrote: > > On Wed, May 18, 2016 at 4:57 AM, Alessio Stalla > <alessiostalla@gmail.com> wrote: >> >> The stance on packages found in the mentioned Quora post is >> based on the old misconception about packages being modules, >> or "software packages" in the Linux distribution sense. >> They're not. They are really just namespaces, containers of >> symbols. So it does not make any sense, to me, that they have >> versions. Software has versions, not names. >
> Systems are what should get versioned but systems are not what > are referred > to by source files. Systems largely coordinate the compilation > and loading of files. Source files generally have no notion of > a system, actually, which may be what is ultimately > problematic. Packages are referred to by source files, > however. When I said packages, I meant it, and it was not a > conflation with the notion of a system. > Why should source files know about systems? Why should packages know about systems?
Maybe they should, maybe they shouldn't. But the crappy packages-as- namespaces system isn't cutting it for large programs, and modules usually naturally have an associated namespace, but there's no formal correspondence between packages and systems. But more often than not there is an implicit association, judging by how most modern Common Lisp libraries are written. So maybe there's value in taking advantage of that correspondence.
> At the companies I have worked in, we rake in 100s of systems, > which consist > of n*100s of systems where n is the average number of packages > per system. These version collisions happen enough, and we are > lucky when they're not silent. > Was it in CL? Was it all in the same process? CL is not designed for these kinds of things.
CL in the same process.
CL very clearly wasn't designed for a time where 100s of packages would be available and simultaneously usable.
But I'm not ready to concede and say "well, Lisp is only good for five or six massive packages or systems written by a few guys, like the good ol' days".
> Often enough, it's an extremely subtle issue that system A > depends > on {system B at time T1} and system C depends on {system B at > time T2}, where T1 and T2 correspond to different versions of > system B, even if not explicitly labeled as such. > A and C are incompatible. You need to fix A, B and/or C. The build system can record this incompatibility and issue an early error when you try to build them together. It can't fix A, B and/or C for you.
A and C are incompatible only because of this whole thing we are talking about. Obviously they're incompatible. That's the whole problem. Saying "Fix A, B, C" is, to me, declaring that this whole thing isn't a problem, and the problem is library writers. Why can't A and C depend on whatever they want at some time? They of course can, but it's non-sense to think people will rename their namespaces every iteration of their software. That's just not how Lisp code has been written.
> Let's suppose we simplify the problem however and assume that B > has been > properly versioned in time, and systems A and C refer properly > to the correct version of system B as a dependency. Even then, > they will not be able to be simultaneously loaded in a > functionally correct fashion. (They *can* both be loaded, but > you'll do some clobbering of state along the way.) > No. The build system can error out and tell you the easy way that you have to fix your bugs before you discover it the hard way
The build system erroring would be a step above the status quo. But maybe there's not actually a logical error. We are only considering it an error because of the way packages work in Lisp, no?
> Whether we think about modules as systems or as packages or as > whatever, the > problem would need to be addressed at the package level, if we > are not assuming that we can't synchronize the universe of > people writing Common Lisp. > The packages have nothing to do with it, unless you fork B into B1 and B2 (or B1 and B, or B and B2).
> Overall, I'd summarize as this. Whether we have the technology > in the Common Lisp ecosystem to accommodate this problem or > not, it's my opinion and the opinion of many of my colleagues > that there is a problem for development teams making extensive > use of open and closed source code. There definitely is not a > discipline that you see with other languages and their > ecosystems (JS/npm, Clojure, etc.) about proper versioning of > systems and furthermore proper loading of the proper versions. > I don't think that's debatable. The closest we have to this > solution is relying on the global nature of a particular > Quicklisp distribution, but not all Lisp software is free and > open source, and even software within Quicklisp has tons of > implied version dependencies. > This is not a very Lisp-specific problem. This is a problem with building software in general. The correct approach is to fix incompatibilities, not to try to deny their existence only to find out the hard way you can't.
Google Bazel provides a good way to deterministically build software that follows the correct discipline. So does NixOS. There may be other such tools. Use them.
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
I don't know. The original question I was answering was why Lisp isn't what we are all using today, and I think this weird thread serves as additional evidence.
Am I way off base here?
Cheers,
Robert
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
[...]
2. There's something like a SocialProblem (TM) where Lispers lack (a) versioning discipline, (b) dependency versioning discipline which exacerbates issues like inadvertent multiple loading, or incompatible loading of libraries.
Semantic versioning won't help you without automatic tools to determine if the API of a library changed in an incompatible way, e.g. if a function removed a keyword arg, or changed the type of an existing arg.
I would say that semvers should in principle help you detect incompatibilities, but as you hinted with "illusion", maybe these in practice can't be trusted or relied upon generally.
They can't be trusted except for trivial libraries because authors in practice don't catch all cases where they break the API. The more expressive the language, the more ways to break the API/ABI. For C/Java, it's actually pretty easy: http://abi-laboratory.pro/tracker/timeline/openssl/.
3. There's actually no issue in principle: inconsistencies need to "just" get fixed by the library vendor/user, or better tooling needs to be used to track your project's direct and transitive dependencies.
It's my personal opinion, as is evident in previous messages, that I think 1 & 2 are issues.
You didn't give an example of how exactly those two things are a problem, and how adding those features would solve the problem(s).
#1: Assuming that there isn't incompatible passing of data, if A # depends on B1 and C depends on B2, and in isolation A-B1 # works/passes tests, and C-B2 works/passes tests, then A-B1 and C- # B2 loaded together (assuming a non-existent Lisp) should work # fine. The problem being addressed is lack of synchrony in the # universe of Lisp modules.
(My first-order thought is that even incompatible data wouldn't be an issue if those data types are properly encapsulated in classes or structures, by virtue of the fact that those types become distinct by way of different names. I envision that only untyped things could cause problems, like [ap]-lists, strings, etc.)
It's easy to come up with such a case, where a library: *) switches from CL epoch timestamps(1900-based) to Unix epoch(1970) *) switches from Unix namestrings to URIs *) switches from relative CL pathnames with implicit base to absolute pathnames. This is properly encapsulated according to your critera In each of these cases, the single versions pass their own internal (unit) tests, but they do not integrate.
#2: If there was diligence is properly versioning libraries, and # diligence in specifying dependency version requirements, then # detection of issues could happen. This is solving recognition of a # possible issue.
Regardless of language, you cannot expect diligence in the open-source world because most libraries are written by people in their free time. I'd love to be paid to work on open-source libraries, and be able to take the time to write design docs, have them reviewed, then write code with unit tests, performance regression tests, etc... Except for a few cases of high-profile projects(Linux, OpenStack), that's unrealistic. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
I’m a bit puzzled by this discussion… Packages are only namespaces and have zero information about semantic protocols. I’m thinking about analogies from OCaml and the ancient COM/OLE interface system. Those examples provide versioning regarding semantic capabilities of a module, and versioning is important so that usage gets tied to a compatible library module. But namespaces are too thin to provide any of this sort of protection. Seems like the discussion needs to address a module / interface system in Lisp, which does not exist to my knowledge. - DM
Dear Robert, you seem to be complaining about issues inherent in any shared namespace, where names can clash. There is no "solution" to that. That's a *whole point* of a namespace that names may be bound to different values, and that incompatible values *shall* clash. Not a bug, a feature! One could imagine that with real first-class namespaces, it should be easier to implement dynamic binding of such names, static binding in lexical scopes, virtualization, etc. Thus, within a same image you could have several binding-sets, and though every binding-set contains compatible bindings, different binding-sets could be mutually incompatible without problem (as long as you don't try to have data from one binding-set interoperate with incompatible code from another). Happily or unhappily, CL has no such provision in its package system. If you want to develop some kind of "better module system" on top of CL, good luck with that: you will have to not only create something that is indeed technically better, but you thereafter have to convince other people to use it so you may benefit from it — and you'll probably have to start creating a massive body of work using your system before people even look at it. I fear the global side-effectful nature of CL will make it very hard to keep anything clean on top of CL, but I'm looking forward to what you have to offer. Personally, I'll say that lisp-interface-library (aka lil) was started as a demo of how you could write more modular code in CL. By modular, here, I mean that you can define interfaces and multiple implementations of those interfaces, what more with parametric polymorphism as well as ad hoc polymorphism. While I think of lil as a technical success, I certainly failed to produce a massive amount of software using it, and have therefore not gotten many people interested in extending it (though I have a handful of contributors). Such interfaces offer a solution to the problem of allowing the peaceful coexistence of multiple instances of solutions to an existing problem. They of course cannot solve the issue of these multiple solutions claiming the same name; if you want multiple versions of an interface implementation, they need different names, though you're welcome to use some version as part of the names. PS: If you have concrete ideas on how to improve ASDF and/or Quicklisp, there are more specialized mailing-lists and/or bug tracking systems for that. For instance, you may propose some kind of off-the-side version tracking and version incompatibility tracking to ASDF, to express constraints that you can't retroactively include in the systems themselves. PPS: You might be interested in reading my blog post about build systems, especially the part discussing programming in the large. http://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems/ —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org As for poverty, no one need be ashamed to admit it: the real shame is in not taking practical measures to escape from it. — Perikles, speaking about the poverty of CL regarding modularity solutions
On May 19, 2016, at 9:01 AM, Stelian Ionescu <sionescu@cddr.org> wrote:
* Because library distribution wasn't a relatively solved problem until recently in Lisp history, some library authors unfortunately include copies of Lisp libraries as a part of their library tree.
That's very rare, in my experience. Do you have examples ?
http://emergent-languages.org/Babel2/ <http://emergent-languages.org/Babel2/> Not sure if we should classify it as a library or an application/system, but a system can often be taken as a library too, right? Best, ---- Alexandre Rademaker http://arademaker.gtihub.com http://researcher.ibm.com/person/br-alexrad
2016-05-19 8:41 GMT+02:00 Robert Smith <quad@symbo1ics.com>:
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
The problem is not a Lisp problem. It occurs in many language ecosystems, and it cannot generally be solved. Things are relatively simple if you just look from the top to the bottom, from the usage of a lower-level library by a higher level library or application functionality. Such usage could confined to within that tree of a linked application, e.g. by symbol or package renaming or the like, and similar techniques have been tried (in Lisp, but also in JVM languages and most likely elsewhere). The real issue though is data types defined by versioned libraries. Suppose that an instance of a data type defined in library A version 1 is passed, through higher levels of the software, to version 2 of that same library. How would one ever expect that to work? Faré is completely right: There is no way to apply some external magic to a hairball of different library versions of libraries and expect the result to work. The only way to deal with it is to actually address the conflicting versions and make sure that all libraries in a system are prepared to use the same overall set of dependencies. If a tool can help with detecting conflicts, it is great, and maybe that is what would be a good goal in the CL world to create such a tool and version tagging infrastructure. I can report that in the Clojure world, every conference that I visited has at least a lightening talk on the subject of library versions and how to deal with them. I have yet to see one where the presenter claims to have a solution, and most of these talks end up concluding that there is a possible problem which cannot generally be solved, but that many Clojure shops get away ignoring the problem. We have been using the leiningen build tool to help us finding library version conflicts and generally strive towards smaller systems to reduce the number of dependencies that we need to synchronize, version-wise. -Hans
Indeed, the problem is present in popular languages used to write big systems comprising hundreds of dependencies and thousands of packages (namespaces). That is, Java, in my experience. The build system (Maven) does nothing to mitigate it; you have to resolve conflicts manually. Sometimes a very commonly used utility library's API is updated in a backwards-incompatible way. In a big project, both versions of the API are likely to be present due to transitive dependencies. You know how do they (the community) address this issue? They rename packages appending a version number. Look at Apache Commons Lang, for example ( https://commons.apache.org/proper/commons-lang/). And, the library maintainer has to do it, or the users of the library have to fork it and maintain their fork. At least, in Lisp we got rename-package, so that the library user *can *provide their alternative package names, even if it's not always the most sensible thing to do. And that only might cut it with utility libraries. In general, you cannot use together two versions of a web framework, ORM, dependency injection container, or any other important piece of your architecture. Nobody expects Java to fail because of that or to ever fix the problem. Nobody thinks the Java community is hostile if they say "we do it manually, deal with it". (Actually, there *is *a solution in Java for that problem, it is called OSGi, and I can assure you that it requires more manual fiddling and it causes more headaches than the problem it is supposed to fix). Yes, that sucks. But not *that* much. Yes, one always expects Lisp to be better and to solve all of the world's problems. Sometimes it doesn't. But it's still great ;) On 19 May 2016 at 09:58, Hans Hübner <hans.huebner@gmail.com> wrote:
2016-05-19 8:41 GMT+02:00 Robert Smith <quad@symbo1ics.com>:
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
The problem is not a Lisp problem. It occurs in many language ecosystems, and it cannot generally be solved.
Things are relatively simple if you just look from the top to the bottom, from the usage of a lower-level library by a higher level library or application functionality. Such usage could confined to within that tree of a linked application, e.g. by symbol or package renaming or the like, and similar techniques have been tried (in Lisp, but also in JVM languages and most likely elsewhere).
The real issue though is data types defined by versioned libraries. Suppose that an instance of a data type defined in library A version 1 is passed, through higher levels of the software, to version 2 of that same library. How would one ever expect that to work?
Faré is completely right: There is no way to apply some external magic to a hairball of different library versions of libraries and expect the result to work. The only way to deal with it is to actually address the conflicting versions and make sure that all libraries in a system are prepared to use the same overall set of dependencies. If a tool can help with detecting conflicts, it is great, and maybe that is what would be a good goal in the CL world to create such a tool and version tagging infrastructure.
I can report that in the Clojure world, every conference that I visited has at least a lightening talk on the subject of library versions and how to deal with them. I have yet to see one where the presenter claims to have a solution, and most of these talks end up concluding that there is a possible problem which cannot generally be solved, but that many Clojure shops get away ignoring the problem. We have been using the leiningen build tool to help us finding library version conflicts and generally strive towards smaller systems to reduce the number of dependencies that we need to synchronize, version-wise.
-Hans
19.05.2016, 11:00, "Hans Hübner" <hans.huebner@gmail.com>:
The real issue though is data types defined by versioned libraries. Suppose that an instance of a data type defined in library A version 1 is passed, through higher levels of the software, to version 2 of that same library. How would one ever expect that to work?
Can that happen without us on the higher level knowingly passing object created by one API, to another, incompatible API, which doesn't accept such objects? I was thinking about that too, but now it appears to me that if incompatible changes in API are marked as such (e.g. by giving the API new name), we always know that the second API is incompatible and can not accept this object. Do you have examples? It would be interesting to explore this problem further. Best regards, - Anton
2016-05-22 13:33 GMT+02:00 Anton Vodonosov <avodonosov@yandex.ru>:
The real issue though is data types defined by versioned libraries. Suppose that an instance of a data type defined in library A version 1 is
19.05.2016, 11:00, "Hans Hübner" <hans.huebner@gmail.com>: passed, through higher levels of the software, to version 2 of that same library. How would one ever expect that to work?
Can that happen without us on the higher level knowingly passing object created by one API, to another, incompatible API, which doesn't accept such objects?
I was thinking about that too, but now it appears to me that if incompatible changes in API are marked as such (e.g. by giving the API new name), we always know that the second API is incompatible and can not accept this object.
Do you have examples? It would be interesting to explore this problem further.
Let's assume that we're using, say, an XML library that can parse dates into instances of a date library, and a database library that accepts instances of date objects of said library for insertion into a database. If the XML library and the database library depended on different versions of the date library and it'd be allowed to have two versions of the date library in one running lisp image, we'd end up passing date instances created by the XML library to the database library. -Hans
In the approach I advocate, this works without problem. If the two versions of date library are incompatible, say datelib and datelib2, then we in principle can not pass a value produced by xmllib:parse-date to dblib:save-object(date2-obj) because these objects are of different type. Application author knows he can't do this. If the two versions of the date library are compatible - the same API, then module loader should only load one version. Or you mean evolution of libraries. When dblib initially depended on datelib, and then changed the dependency to datelib2 ? In this case the _public_ API of dblib changes - yesterday dblib:save-object was accepting datelib:date object, and today it accepts datelib2:date. So new version of dblib should become dblib2. As long as application author uses old dblib, everything is compatible. When the application author wants to use dblib2 he is aware that is different API and he sees that the function dblib2:save-object can't accept the date value he has. 22.05.2016, 14:53, "Hans Hübner" <hans.huebner@gmail.com>:
2016-05-22 13:33 GMT+02:00 Anton Vodonosov <avodonosov@yandex.ru>:
19.05.2016, 11:00, "Hans Hübner" <hans.huebner@gmail.com>:
The real issue though is data types defined by versioned libraries. Suppose that an instance of a data type defined in library A version 1 is passed, through higher levels of the software, to version 2 of that same library. How would one ever expect that to work?
Can that happen without us on the higher level knowingly passing object created by one API, to another, incompatible API, which doesn't accept such objects?
I was thinking about that too, but now it appears to me that if incompatible changes in API are marked as such (e.g. by giving the API new name), we always know that the second API is incompatible and can not accept this object.
Do you have examples? It would be interesting to explore this problem further.
Let's assume that we're using, say, an XML library that can parse dates into instances of a date library, and a database library that accepts instances of date objects of said library for insertion into a database. If the XML library and the database library depended on different versions of the date library and it'd be allowed to have two versions of the date library in one running lisp image, we'd end up passing date instances created by the XML library to the database library.
-Hans
On Sun, 2016-05-22 at 14:33 +0300, Anton Vodonosov wrote:
19.05.2016, 11:00, "Hans Hübner" <hans.huebner@gmail.com>:
The real issue though is data types defined by versioned libraries. Suppose that an instance of a data type defined in library A version 1 is passed, through higher levels of the software, to version 2 of that same library. How would one ever expect that to work?
Can that happen without us on the higher level knowingly passing object created by one API, to another, incompatible API, which doesn't accept such objects?
I was thinking about that too, but now it appears to me that if incompatible changes in API are marked as such (e.g. by giving the API new name), we always know that the second API is incompatible and can not accept this object.
Do you have examples? It would be interesting to explore this problem further.
I gave a few in a previous message here: It's easy to come up with such a case, where a library: *) switches from CL epoch timestamps(1900-based) to Unix epoch(1970) *) switches from Unix namestrings to URIs *) switches from relative CL pathnames with implicit base to absolute pathnames In each of these cases, the single versions pass their own internal (unit) tests, but they do not integrate. If libraries change the semantics of slots whose types are standard CL types, you can't even detect this without extensive integration tests. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
22.05.2016, 17:18, "Stelian Ionescu" <sionescu@cddr.org>:
It's easy to come up with such a case, where a library: *) switches from CL epoch timestamps(1900-based) to Unix epoch(1970) *) switches from Unix namestrings to URIs *) switches from relative CL pathnames with implicit base to absolute pathnames
Please clarify, switches how? Where?
22.05.2016, 17:36, "Anton Vodonosov" <avodonosov@yandex.ru>:
22.05.2016, 17:18, "Stelian Ionescu" <sionescu@cddr.org>:
It's easy to come up with such a case, where a library: *) switches from CL epoch timestamps(1900-based) to Unix epoch(1970) *) switches from Unix namestrings to URIs *) switches from relative CL pathnames with implicit base to absolute pathnames
Please clarify, switches how? Where?
Because I was asking for examples of what Hans describes: how object created by one version of a library gets passed to another version of the library somewhere on higher level of the software. My point is that I don't see how that can happen, if the the different versions of the library are different APIs.
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
It's not a Lisp-only problem, it's a general boolean satisfiability problem that all languages have. The only way to deal with this is to ensure that your entire ecosystem works with the same set of dependencies, to check-in those dependencies in your repository and not rely on "semantic versioning" or other such illusions, run the tests(do actual QA) and be very conservative with upgrades. Having a source control system that can keep everything in one repository, with partial checkouts, is very useful in achieving that. I've started to really appreciate Perforce in the last year or so. At my previous company, we had so many problems with Node.js(good for prototyping) because of the fact that npm(the package manager) downloaded private copies of the libraries, that in the end I think they rewrote the server to something more sensible, Java. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Thanks all for the rest of your comments. Interesting notes about Clojure, Hans. I agree that this problem is not unique to Lisp, but what is relatively unique is that it's a quiet problem. (Mismatching of transitive dependencies can go unnoticed until something eventually breaks.) The only question in response to many of your relatively-in-agreement-with-each-other comments is: do you think the relevant portions of Lisp, the relevant portions of the Lisp tool chain (ASDF, QL), and the way in which Lisp seems to be popularly used in open source are as a whole close to optimal in what we can do in 2016 to even detect, let alone address, these kinds of issues? Or is manual per-project vetting and curation of libraries the best possible? Robert On Thursday, May 19, 2016, Stelian Ionescu <sionescu@cddr.org> wrote:
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
It's not a Lisp-only problem, it's a general boolean satisfiability problem that all languages have. The only way to deal with this is to ensure that your entire ecosystem works with the same set of dependencies, to check-in those dependencies in your repository and not rely on "semantic versioning" or other such illusions, run the tests(do actual QA) and be very conservative with upgrades. Having a source control system that can keep everything in one repository, with partial checkouts, is very useful in achieving that. I've started to really appreciate Perforce in the last year or so.
At my previous company, we had so many problems with Node.js(good for prototyping) because of the fact that npm(the package manager) downloaded private copies of the libraries, that in the end I think they rewrote the server to something more sensible, Java.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Thanks all for the rest of your comments. Interesting notes about Clojure, Hans.
I agree that this problem is not unique to Lisp, but what is relatively unique is that it's a quiet problem. (Mismatching of transitive dependencies can go unnoticed until something eventually breaks.)
That's a problem with all languages that don't do static types. Node.js, Ruby, Python are all afflicted by this issue. Even with static types, that doesn't solve other types of problems, like a dependency that switched from O(n^2) to O(n*log(n)) implementation in a method, that exposed a race condition in the user code and caused the server to crash in production during the Black Friday peak.
The only question in response to many of your relatively-in-agreement-with-each- other comments is: do you think the relevant portions of Lisp, the relevant portions of the Lisp tool chain (ASDF, QL), and the way in which Lisp seems to be popularly used in open source are as a whole close to optimal in what we can do in 2016 to even detect, let alone address, these kinds of issues? Or is manual per-project vetting and curation of libraries the best possible?
There are ways, one of which would be a very strict build system like Bazel. That has several problems of its own, though. As somebody noticed in a blog post about the latest ELS, Bazel has a large overhead and is cumbersome for small projects.
Robert
On Thursday, May 19, 2016, Stelian Ionescu <sionescu@cddr.org> wrote:
__
I don't know much about Bazel, and I know a little about NixOS. Regardless, this seems to be moving the problem into how we globally synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs into when writing Common Lisp code, and not an issue that one runs into writing in another language and associated ecosystem. To me, that's a Lisp problem. We can do some creative academic definitions, I think, but it's a problem when choosing Lisp as a tool.
It's not a Lisp-only problem, it's a general boolean satisfiability problem that all languages have. The only way to deal with this is to ensure that your entire ecosystem works with the same set of dependencies, to check-in those dependencies in your repository and not rely on "semantic versioning" or other such illusions, run the tests(do actual QA) and be very conservative with upgrades. Having a source control system that can keep everything in one repository, with partial checkouts, is very useful in achieving that. I've started to really appreciate Perforce in the last year or so.
At my previous company, we had so many problems with Node.js(good for prototyping) because of the fact that npm(the package manager) downloaded private copies of the libraries, that in the end I think they rewrote the server to something more sensible, Java.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Versioning packages is very simple. Suppose we have a package: (defpackage :mylib ...) Now we want new version of this package. We do: (defpackage :mylib2 ...) 18.05.2016, 07:23, "Jean-Claude Beaudoin" <jean.claude.beaudoin@gmail.com>:
Hi CL Pros,
I just came across a post on Quora by Robert Smith where he mentions the idea of assigning some sort of "version" to Common Lisp packages in order to improve code configuration control.
This seems to me to be a pretty neat and interesting idea!
Has anyone of you explored such a concept, or know of anyone that did? Or, is anyone of you curious about it?
I think pretty seriously that I will implement something like this as a CL extension in my upcoming MKCL 2.0.
Cheers,
Jean-Claude Beaudoin
participants (11)
-
Alessio Stalla -
Alexandre Rademaker -
Anton Vodonosov -
David McClain -
Faré -
Hans Hübner -
Jean-Claude Beaudoin -
Pascal Bourguignon -
Robert Smith -
Scott L. Burson -
Stelian Ionescu