Hi!
Every time I read "asdf", I feel a pain. I've read that there is an attempt to gain resources to improve asdf. I have a sort of plan.
1. Shims. Recent tightening of rules for system definitions is ok, but there are old systems with no maintainers. If such system does not obey the rules, one can introduce "shim" concept. I've met them in JS culture where they serve as third-party adapters to connect two mismatching things.
In the simplest way shim is just an alternative directory hierarchy with shim asd files, isomorphic to local lisp directory structure.
When looking for system, asdf must search in shims directory first, and only then in the directory of the file itself. Also things like quicklisp might take care of installing shims where they exist. Maintanence of shims for all popular systems can be done within a separate git repository.
2. Get rid of upgrade. Upgrade feature requires to maintain a 3d array of possible cases, where dimensions are "old asdf version", "new asdf version" and "lisp implementation". It is hard to maintain and it will get harder and harder to maintain as the time goes on. Also upgrade is a good test of CLOS, but running tests at the very beginning of image bootstrap is not a good idea because there is no e.g. SLIME to work with convenience.
3. Last, but most important actually. Prioritize manual, FAQ, Wiki and all like this.
Instead, when loading asdf, allow the user to pass the parameter that fills asdf database with the initial loaded system information. It would be also good to have an utility to extract this database from some old asdf versions. This way we have a slight chance to make things easier, upgrade process explicit and under user's control.
On 19 Feb 2019, at 1:18, 73budden wrote:
Hi!
Every time I read "asdf", I feel a pain. I've read that there is an attempt to gain resources to improve asdf. I have a sort of plan.
- Shims. Recent tightening of rules for system definitions is ok, but
there are old systems with no maintainers. If such system does not obey the rules, one can introduce "shim" concept. I've met them in JS culture where they serve as third-party adapters to connect two mismatching things.
In the simplest way shim is just an alternative directory hierarchy with shim asd files, isomorphic to local lisp directory structure.
When looking for system, asdf must search in shims directory first, and only then in the directory of the file itself. Also things like quicklisp might take care of installing shims where they exist. Maintanence of shims for all popular systems can be done within a separate git repository.
For the moment, I think the only issue related to rules tightening is that we ask people to use the proper complex system names (see my recent change to the ASDF manual, available from the page on `common-lisp.net`. Is that what you are referring to? Or are there other issues causing pain?
If it's just that, insert the name of the asd file (name only -- no extension) into `*known-systems-with-bad-secondary-system-names*`. E.g.,
``` (setf (gethash "terrible-legacy-system" *known-systems-with-bad-secondary-system-names*) t) ```
... and that will suppress the warnings for you.
That set has "cl-ppcre" built into it, and I'm happy to add other systems that are in common use, but unmaintained.
I should put documentation of this into the manual....
The shims idea is interesting, but isn't it a lot of trouble to make and maintain a shim (which has to somehow indicate where to find the source files for the library), compared to forking a system?
If you have a more specific idea for how this works, I'd be happy to discuss it further.
- Get rid of upgrade. Upgrade feature requires to maintain a 3d array
of possible cases, where dimensions are "old asdf version", "new asdf version" and "lisp implementation". It is hard to maintain and it will get harder and harder to maintain as the time goes on. Also upgrade is a good test of CLOS, but running tests at the very beginning of image bootstrap is not a good idea because there is no e.g. SLIME to work with convenience.
I don't know about doing away with upgrade, but certainly cutting **way** down on the upgrades we support is a good idea. For CMUCL, we now only test and support 3.3.0+
I'm willing to do that for many more implementations, if I can figure out what ASDF versions they are shipping on what target platforms.
- Last, but most important actually. Prioritize manual, FAQ, Wiki and
all like this.
I'm ok with that, but I don't have the time for an end-to-end rewrite of the manual. Note that I am also going to integrate **everything** into the manual, because I don't have the time to support more than one documentation set (e.g., Fare's Markdown stuff will get folded into the manual).
Part of the problem is that the manual doesn't have a clear structure. It shoold somehow start off by teaching only the basics, in a very brief discussion, then get to more advanced topics, then the object model, and stuff that only an extender, developer, or maintainer would want to know. But it's quite possible I will retire or die before I manage to do that level of rewrite.
Restructuring is *really* not something that Texinfo makes easy, but Texinfo seems to be the worst documentation system ... except for all the other ones.
A contribution I would *love* to have is an adaptation of the SBCL manual code that extracts symbols and docstrings for inclusion in Texinfo. At one point I looked into Didier's library (DecIt, I think?), but while it's good at extraction, I couldn't see how what it extracts could be moved around and integrated with large blocks of text.
I'm happy to accept FAQ suggestions and get them into the manual.
Instead, when loading asdf, allow the user to pass the parameter that fills asdf database with the initial loaded system information. It would be also good to have an utility to extract this database from some old asdf versions. This way we have a slight chance to make things easier, upgrade process explicit and under user's control.
I'm not exactly sure what is meant by the above.
I have a number of different projects I work on, each with their own set of libraries. I have made simple functions for each of these projects that populates `asdf:*central-registry*`, and when I sit down to work on one of these systems, I just invoke that function. That's always been sufficient for me. So something like:
``` CL-USER> (shop2) "SHOP2 libraries available." NIL CL-USER> (asdf:test-system "shop2") .... ```
I still use `*central-registry*` for that because (1) I wrote it before Fare's new DSL was available and (2) I really don't have a handle on how to debug DSL-based configuration when it goes wrong (and configuration by environment variable is even more difficult, because the debugging environment isn't available until after the environment variables have been processed).
I hope this is helpful, R
: 73budden
: Robert Goldman
- Shims. [...]
The shims idea is interesting, but isn't it a lot of trouble to make and maintain a shim (which has to somehow indicate where to find the source files for the library), compared to forking a system?
Forking seems more robust, too. The problem with shims is that they quickly get out of date. Or they must be paired with a specific revision of the system being built, like the build_defs in the bazel lisp support. https://github.com/qitab/bazelisp
- Get rid of upgrade.
Upgrade is very important. See my original ASDF article: https://fare.livejournal.com/149264.html Upgrade is even more important since implementations often include an old version of ASDF: CCL has 3.2.0, LispWorks has 3.1.7 (IIRC), and sometimes SBCL isn't upgraded for years. At least, unlike in the bad old ASDF 1 days, there is now a meaningful version to query.
- Last, but most important actually. Prioritize manual, FAQ, Wiki and
all like this.
I'm ok with that, but I don't have the time for an end-to-end rewrite of the manual. Note that I am also going to integrate everything into the manual, because I don't have the time to support more than one documentation set (e.g., Fare's Markdown stuff will get folded into the manual).
I don't love Markdown, but I resent that the texinfo version forces the authors to manually maintain an index of nodes, etc. This makes refactoring the structure of the manual particularly unpleasant. Hopefully, Robert finds some other format that does the indexing for you. I recommend Racket Scribble, but there are no doubt other choices, too. My own Exscribe works too but only to produce HTML (its PDF backend is too limited).
Part of the problem is that the manual doesn't have a clear structure. It shoold somehow start off by teaching only the basics, in a very brief discussion, then get to more advanced topics, then the object model, and stuff that only an extender, developer, or maintainer would want to know. But it's quite possible I will retire or die before I manage to do that level of rewrite.
The "best practices" document kind of took that approach of layering sections by skill. It is far from complete with that approach, though. But maybe a merge of it with the current manual can help.
A contribution I would love to have is an adaptation of the SBCL manual code that extracts symbols and docstrings for inclusion in Texinfo. At one point I looked into Didier's library (DecIt, I think?), but while it's good at extraction, I couldn't see how what it extracts could be moved around and integrated with large blocks of text.
Scribble and Exscribe being programmable mean you could do whatever you want here.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org One man alone can be pretty dumb sometimes, but for real bona fide stupidity, there ain't nothing can beat teamwork. — Edward Abbey
The shims idea is interesting, but isn't it a lot of trouble to make and maintain a shim (which has to somehow indicate where to find the source files for the library), compared to forking a system?
+1
there's already the sharplispers group for taking over unmaintained systems (not to be confused with hostile systems):
https://github.com/sharplispers
Hello! Thanks for the feedback. Actually I don't use lisp for about a half a year now, and looking for a new job, so I definitely have no time to dig into much detail. But I was keeping those shims in mind for about a year, so I thought now it is a good time to share thoughts. I'll try to clarify some of my points.
1. Overall ideology. Lisp community is shrinking. Keeping things simple and sustainable is a key to survival. In asdf I see the accumulation of complexity with no end. That is bad generally.
2. Shims. If the library is dead (unmaintained), there is no effort maintaining shims once they are created, except if asdf 4 appear that breaks compatibility. If anyone (alive) uses the lib, he anyways have to deal with the broken system which does not build. By introducing shims, we create a way to share and re-use this work. The key here is that shim is alive (can be maintained, e.g. if asdf4 appear sometimes) while the library is dead. Forking all libs is also the way to go, but it has issues, at least in case of github. Fork must be independent (have no reference to the previous work, but be loaded into github independently), otherwise there is a mess. With no access to original lib there is no way to say "this library is dead, and that fork is a preference". Forks by themselves are not searchable. Entry point is the original lib, not the fork. So while searching for any lib, one has to study the fork network and guess which library is the right one, which is heuristical and error-prone (there is no way to distinguish someone's personal experiments from the mainstream fork). If there is a lisp-specific index of mainstream forks, then of course it can be maintained instead of shims. Quicklisp is an obvious place for that. Maybe it is a better than shims (no need to change asdf), but then github search is no more an entry point to find lisp libs. Maybe it is fine, I didn't think about it.
Having a special key which disables warnings does not look as a good decision, because it adds more complexity (not only in the implementation, but also in the use of the asdf) and preserves the old mess. So the goal to get rid of that misnamed systems is not achieved at all.
3. Uprgrade. I know there are vendor-specific asdf versions, but it does not matter. What we need from old asdf is its database converted to the new format, no need to do hot upgrade. There is no problem to specify two things in a config file - a path to a converted database, saved as a file, and a path to a new asdf, so we don't need things like old-asdf:load-system :new-asdf. Just don't use old asdf to load new one, use simple "load", which will destroy old asdf package, create a new one and load the database from the file. That's it. We only need a converted for each old asdf version which exports the database to the file. This way asdf code for old and new versions is decoupled completely, and the upgrade process simplifies dramatically. As far as I can imagine, asdf database structure is very simple. But if the implementation contains a set of asdf files in an obsolete format, then implementation can be broken after the upgrade, Current approach does not solve this either: new asdf must understand old asdf files also. Here again we come to the idea of shims for implementation's asdf files, which can be stored externally if it is impossible to promote changes to the implementation.
Sorry I have no resources to participate in the implementation, but if I return to the lisp universe at some point, I'll maybe start with something like this.
Best Regards!
2019-02-20 4:57 GMT+03:00, Attila Lendvai attila@lendvai.name:
The shims idea is interesting, but isn't it a lot of trouble to make and maintain a shim (which has to somehow indicate where to find the source files for the library), compared to forking a system?
+1
there's already the sharplispers group for taking over unmaintained systems (not to be confused with hostile systems):
https://github.com/sharplispers
-- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Tact is a skill that can turn brutal honesty into just honesty. It's a skill that develops with practice, and one that's harder to use when emotions are running high. But you can't go towards someone with a verbal fist and expect a hug in return. When method matches intention, outcomes are much more peaceful.” — Doe Zantamata
simple and sustainable is a key to survival. In asdf I see the accumulation of complexity with no end. That is bad generally.
old ASDF was broken in countless ways, and i've experienced the bugs in my everyday activities.
the question is whether ASDF has any accidental complexity, i.e. complexity that is not inherent to the problem domain, and thus could be eliminated -- at least theoretically.
-
73budden -
Attila Lendvai -
Faré -
Robert Goldman