Dear all,
I hope that 2.008 will be my last ASDF release. I may still make emergency bug fixes (if any is needed), or merge patches sent to me, but I don't intend to actively develop ASDF anymore. I feel it has reached the point where I wanted it to be, although it took much more efforts than I feared. Just look at the git log to see how hard it was, and read my and Robert's paper for ILC'2010 to get an idea why we did things that way. http://common-lisp.net/project/asdf/ilc2010draft.pdf
And so, ASDF is looking for a new active maintainer. To volunteer, just start hacking on your own repo, and I'll hand you the keys after I merge your first commit.
I intend to focus on XCVB, and its dependencies, cl-iolib and libfixposix. If ASDF moves towards more declarative .asd files, XCVB will certainly try to be compatible.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] [...] there is what I call the "roundtrip fallacy": it is a mistake to use, as journalists and some economists do, statistics without logic but the reverse does not hold: It is not a mistake to use logic without statistics. — N. C. Taleb, Fooled by Randomness, 2004.
Hi,
I have spent some time this year familiarizing myself with ASDF, and afterwards I think, that it's a great build-tool (possibly, the best one around, actually), which has a lot of hidden potential. So I'd be willing to help as the maintainer or one of the maintainers.
But there's more to maintaining ASDF, than to other projects, as it is a critical part of every Lisp environment. That's why I think it would be best to have a separate Release manager (performing most duties of maintainer), and separate people, who will take decision about feature requests. There are two possible models here: benevolent dictator and committee. Both have their drawbacks, but dictator model can only work with the original author, so it's not applicable here. Speaking about committee model, I think that it should be formed by the representatives of most of all Lisp implementations, as well as Release manager and possibly other people who were heavily involved in ASDF development, all having some kind of a veto power. It can be backed by such simple thing as just a separate mailing list for feature requests or an Issue tracker.
As a Release manager I'd do the following: - establish a regular release cycle (bi-monthly), transition to follow the Rational version policy - finish creating a comprehensive test-suite, and organize some automatic testing process - move development to github, where there is some rather convenient infrastructure, like issue tracking (leaving a mirror on common-lisp.net) - work on improving documentation (also I'm currently doing a series of articles about ASDF in Russian in my blog http://lisp-univ-etc.blogspot.com, that I'm also going to translate to English eventually) - continue work on separating ASDF itself and some support subsystems - establish some basic contribution guidelines - answer questions in the mailing list - contribute to bug-fixing
My contribution to the Lisp library world is not that big. It can be seen at http://github.com/vseloved, so I'm not an expert Lisp developer (although I claim to be one in my CV :) Yet, I think I understand ASDF and it's peculiarities enough, and also have a vision of its future to perform that role. So I'm willing to do that, if no better candidate will volunteer.
Speaking about future vision, I think we should transition to a completely declarative (read, but not eval'd) .asd-file format. Also currently (partially) broken things, like support for versioning and forcing should be fixed. Actually, that is mostly all, that is lacking. (Surely, other needs might actualize in the future...) What will be left is just to comprehensively and concisely describe all the patterns of possible ASDF usage.
Best regards, Vsevolod Dyomkin
On Sat, Sep 11, 2010 at 8:41 PM, Faré fahree@gmail.com wrote:
Dear all,
I hope that 2.008 will be my last ASDF release. I may still make emergency bug fixes (if any is needed), or merge patches sent to me, but I don't intend to actively develop ASDF anymore. I feel it has reached the point where I wanted it to be, although it took much more efforts than I feared. Just look at the git log to see how hard it was, and read my and Robert's paper for ILC'2010 to get an idea why we did things that way. http://common-lisp.net/project/asdf/ilc2010draft.pdf
And so, ASDF is looking for a new active maintainer. To volunteer, just start hacking on your own repo, and I'll hand you the keys after I merge your first commit.
I intend to focus on XCVB, and its dependencies, cl-iolib and libfixposix. If ASDF moves towards more declarative .asd files, XCVB will certainly try to be compatible.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] [...] there is what I call the "roundtrip fallacy": it is a mistake to use, as journalists and some economists do, statistics without logic but the reverse does not hold: It is not a mistake to use logic without statistics. — N. C. Taleb, Fooled by Randomness, 2004.
asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
On 9/12/10 Sep 12 -11:48 AM, Vsevolod Dyomkin wrote:
Hi,
Some random comments....
I have spent some time this year familiarizing myself with ASDF, and afterwards I think, that it's a great build-tool (possibly, the best one around, actually), which has a lot of hidden potential. So I'd be willing to help as the maintainer or one of the maintainers.
....
As a Release manager I'd do the following:
- establish a regular release cycle (bi-monthly), transition to follow
the Rational version policy
Three points with respect to this:
1. We should probably have a definition of "rational version policy," at least by citation.
2. If the rational version policy is the numbering scheme I found by googling, I don't believe it is compatible with asdf:version-satisfies. I would suggest we avoid adopting a policy that doesn't meet that constraint.
3. Is this bi-monthly release policy wrt ASDF 3? In my fondest dreams ASDF2 is getting stable and the ASDF 2 release cycle should be "very rarely, never in the limit." I say this not just because I am looking forward to ASDF 2 being done, but also because ASDF is critically foundational to the entire open source CL community. In order for members of this community to be able to share work, ASDF's API needs to be very, very stable (see the paper Faré and I wrote about ASDF 2 development for a more thorough presentation of the need for stability).
- finish creating a comprehensive test-suite, and organize some
automatic testing process
This would be very good. BTW, I have just discovered that there are at least three different ways to start CCL, and only one of them is currently exercised by our test suite.
- move development to github, where there is some rather
convenient infrastructure, like issue tracking (leaving a mirror on common-lisp.net http://common-lisp.net)
Is there some reason that common-lisp.net + launchpad is unsatisfactory?
- work on improving documentation (also I'm currently doing a series of
articles about ASDF in Russian in my blog http://lisp-univ-etc.blogspot.com, that I'm also going to translate to English eventually)
- continue work on separating ASDF itself and some support subsystems
- establish some basic contribution guidelines
- answer questions in the mailing list
- contribute to bug-fixing
This sounds great.
Best, Robert
On Sun, Sep 12, 2010 at 8:09 PM, Robert Goldman rpgoldman@sift.info wrote:
On 9/12/10 Sep 12 -11:48 AM, Vsevolod Dyomkin wrote:
Hi,
Some random comments....
I have spent some time this year familiarizing myself with ASDF, and afterwards I think, that it's a great build-tool (possibly, the best one around, actually), which has a lot of hidden potential. So I'd be willing to help as the maintainer or one of the maintainers.
....
As a Release manager I'd do the following:
- establish a regular release cycle (bi-monthly), transition to follow
the Rational version policy
Three points with respect to this:
- We should probably have a definition of "rational version policy,"
at least by citation.
http://docs.rubygems.org/read/chapter/7
- If the rational version policy is the numbering scheme I found by
googling, I don't believe it is compatible with asdf:version-satisfies. I would suggest we avoid adopting a policy that doesn't meet that constraint.
Last time I looked, version-satisfies quite-happily handled exact versions, like "1.2.3", which also comply with the "rational" policy. At the same time, current versions, like 2.103 are actually 2.1.3 in disguise, which is a little annoying. Yet, the biggest benefit of the rational policy (or at least some policy) is that it's predictable.
- Is this bi-monthly release policy wrt ASDF 3? In my fondest dreams
ASDF2 is getting stable and the ASDF 2 release cycle should be "very rarely, never in the limit." I say this not just because I am looking forward to ASDF 2 being done, but also because ASDF is critically foundational to the entire open source CL community. In order for members of this community to be able to share work, ASDF's API needs to be very, very stable (see the paper Faré and I wrote about ASDF 2 development for a more thorough presentation of the need for stability).
Rarely a release should break backwards compatibility. But from what I see now, there's quite a lot of fixes to ASDF (at least a couple each month), and I don't forsee the change to such situation.
- finish creating a comprehensive test-suite, and organize some
automatic testing process
This would be very good. BTW, I have just discovered that there are at least three different ways to start CCL, and only one of them is currently exercised by our test suite.
- move development to github, where there is some rather
convenient infrastructure, like issue tracking (leaving a mirror on common-lisp.net http://common-lisp.net)
Is there some reason that common-lisp.net + launchpad is unsatisfactory?
It's much less integrated and more clumsy (I filed several bugs through launchpad myself and don't like the experience). It seems, that github is a much friendlier place for that, and it is developing pretty fast. Yet, I don't think, that it is critical, so if other contributors would like to stay with the current variant, I don't object.
- work on improving documentation (also I'm currently doing a series of
articles about ASDF in Russian in my blog http://lisp-univ-etc.blogspot.com, that I'm also going to translate to English eventually)
- continue work on separating ASDF itself and some support subsystems
- establish some basic contribution guidelines
- answer questions in the mailing list
- contribute to bug-fixing
This sounds great.
Best, Robert
asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
On 9/12/10 Sep 12 -1:55 PM, Vsevolod Dyomkin wrote:
On Sun, Sep 12, 2010 at 8:09 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:
On 9/12/10 Sep 12 -11:48 AM, Vsevolod Dyomkin wrote: > Hi, Some random comments....
....
2. If the rational version policy is the numbering scheme I found by googling, I don't believe it is compatible with asdf:version-satisfies. I would suggest we avoid adopting a policy that doesn't meet that constraint.Last time I looked, version-satisfies quite-happily handled exact versions, like "1.2.3", which also comply with the "rational" policy. At the same time, current versions, like 2.103 are actually 2.1.3 in disguise, which is a little annoying. Yet, the biggest benefit of the rational policy (or at least some policy) is that it's predictable.
I don't believe that is correct. Here is a line I have ripped from version-satisfies-p, which seems to clearly show that 2.103 is (2 103) not (2 1 3):
ASDF(76): (mapcar #'parse-integer (split-string "2.103" :separator ".")) (2 103)
3. Is this bi-monthly release policy wrt ASDF 3? In my fondest dreams ASDF2 is getting stable and the ASDF 2 release cycle should be "very rarely, never in the limit." I say this not just because I am looking forward to ASDF 2 being done, but also because ASDF is critically foundational to the entire open source CL community. In order for members of this community to be able to share work, ASDF's API needs to be very, very stable (see the paper Faré and I wrote about ASDF 2 development for a more thorough presentation of the need for stability).Rarely a release should break backwards compatibility. But from what I see now, there's quite a lot of fixes to ASDF (at least a couple each month), and I don't forsee the change to such situation.
I certainly hope you are wrong in thinking this won't change, but fine.
I think we are still going to want a release policy for ASDF 3 and ASDF 2 separately, with the ASDF API being stable over ASDF 2 releases.
> - finish creating a comprehensive test-suite, and organize some > automatic testing process This would be very good. BTW, I have just discovered that there are at least three different ways to start CCL, and only one of them is currently exercised by our test suite. > - move development to github, where there is some rather > convenient infrastructure, like issue tracking (leaving a mirror on > common-lisp.net <http://common-lisp.net> <http://common-lisp.net>) Is there some reason that common-lisp.net <http://common-lisp.net> + launchpad is unsatisfactory?It's much less integrated and more clumsy (I filed several bugs through launchpad myself and don't like the experience). It seems, that github is a much friendlier place for that, and it is developing pretty fast. Yet, I don't think, that it is critical, so if other contributors would like to stay with the current variant, I don't object.
I am just feeling the need for some stability, having been through CVS and git already (and the git transition was pretty bumpy). Getting all the information OUT of launchpad and into a different issue tracker seems painful. However, if someone wants to take the trouble of actually porting all the launchpad data, who am I to object?
Best, r
On Sun, Sep 12, 2010 at 10:22 PM, Robert Goldman rpgoldman@sift.infowrote:
On 9/12/10 Sep 12 -1:55 PM, Vsevolod Dyomkin wrote:
On Sun, Sep 12, 2010 at 8:09 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:
On 9/12/10 Sep 12 -11:48 AM, Vsevolod Dyomkin wrote: > Hi, Some random comments........
2. If the rational version policy is the numbering scheme I found by googling, I don't believe it is compatible withasdf:version-satisfies.
I would suggest we avoid adopting a policy that doesn't meet that constraint.Last time I looked, version-satisfies quite-happily handled exact versions, like "1.2.3", which also comply with the "rational" policy. At the same time, current versions, like 2.103 are actually 2.1.3 in disguise, which is a little annoying. Yet, the biggest benefit of the rational policy (or at least some policy) is that it's predictable.
I don't believe that is correct. Here is a line I have ripped from version-satisfies-p, which seems to clearly show that 2.103 is (2 103) not (2 1 3):
ASDF(76): (mapcar #'parse-integer (split-string "2.103" :separator ".")) (2 103)
And that is what's annoying, since actually (semantically) 103 is meant as 1.3, no?
3. Is this bi-monthly release policy wrt ASDF 3? In my fondestdreams
ASDF2 is getting stable and the ASDF 2 release cycle should be "very rarely, never in the limit." I say this not just because I amlooking
forward to ASDF 2 being done, but also because ASDF is critically foundational to the entire open source CL community. In order for members of this community to be able to share work, ASDF's API needsto
be very, very stable (see the paper Faré and I wrote about ASDF 2 development for a more thorough presentation of the need forstability).
Rarely a release should break backwards compatibility. But from what I see now, there's quite a lot of fixes to ASDF (at least a couple each month), and I don't forsee the change to such situation.
I certainly hope you are wrong in thinking this won't change, but fine.
I think we are still going to want a release policy for ASDF 3 and ASDF 2 separately, with the ASDF API being stable over ASDF 2 releases.
> - finish creating a comprehensive test-suite, and organize some > automatic testing process This would be very good. BTW, I have just discovered that there areat
least three different ways to start CCL, and only one of them is currently exercised by our test suite. > - move development to github, where there is some rather > convenient infrastructure, like issue tracking (leaving a mirror on > common-lisp.net <http://common-lisp.net> <http://common-lisp.net>) Is there some reason that common-lisp.net <http://common-lisp.net> + launchpad is unsatisfactory?It's much less integrated and more clumsy (I filed several bugs through launchpad myself and don't like the experience). It seems, that github is a much friendlier place for that, and it is developing pretty fast. Yet, I don't think, that it is critical, so if other contributors would like to stay with the current variant, I don't object.
I am just feeling the need for some stability, having been through CVS and git already (and the git transition was pretty bumpy). Getting all the information OUT of launchpad and into a different issue tracker seems painful. However, if someone wants to take the trouble of actually porting all the launchpad data, who am I to object?
Best, r
ASDF(76): (mapcar #'parse-integer (split-string "2.103" :separator ".")) (2 103)
And that is what's annoying, since actually (semantically) 103 is meant as 1.3, no?
Yup, and it's all my fault for failing to switch numbering during the 2.0 release. Hopefully, 3.0 will fix that.
BTW, I just committed 2.129 to fix one more bug. Sigh. I hope a new maintainer will release that.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Socialists, who want everyone enslaved, don't object to private slave-ownership that it implies slaves, but that it implies non-slaves.
On 9/12/10 Sep 12 -2:57 PM, Faré wrote:
ASDF(76): (mapcar #'parse-integer (split-string "2.103" :separator ".")) (2 103)
And that is what's annoying, since actually (semantically) 103 is meant as 1.3, no?
Yup, and it's all my fault for failing to switch numbering during the 2.0 release. Hopefully, 3.0 will fix that.
Sorry. I thought the implication was that there was some bug in the revision number handling, but I see that's not what was meant.
Honestly, I don't know if we have three significant fields worth of version information, anyway. We have 2 versus nothing or, perhaps soon 3, indicating different APIs. Inside 2 we just have individual release numbers (patch levels).
This closes the door to having an ASDF 2.1 (2.2?), but does anyone really care that much? The next version can just be 3 and then 4....
On 9/11/10 Sep 11 -12:41 PM, Faré wrote:
Dear all,
I hope that 2.008 will be my last ASDF release. I may still make emergency bug fixes (if any is needed), or merge patches sent to me, but I don't intend to actively develop ASDF anymore. I feel it has reached the point where I wanted it to be, although it took much more efforts than I feared. Just look at the git log to see how hard it was, and read my and Robert's paper for ILC'2010 to get an idea why we did things that way. http://common-lisp.net/project/asdf/ilc2010draft.pdf
And so, ASDF is looking for a new active maintainer. To volunteer, just start hacking on your own repo, and I'll hand you the keys after I merge your first commit.
I figure to continue my limited role as assistant maintainer, and general voice in favor of limited modification (call me the "bug-compatibility advocate"), but don't have the time available to do more than that.
But I do have the change bit, which is probably a good thing, since there were some problems early in the history where owing to personnel changeover it was hard to get people promoted to commit status, etc.
best, r
On Sat, Sep 11, 2010 at 7:41 PM, Faré fahree@gmail.com wrote:
I hope that 2.008 will be my last ASDF release. I may still make emergency bug fixes (if any is needed), or merge patches sent to me, but I don't intend to actively develop ASDF anymore
Thanks for all your work up to now, Fare.
And so, ASDF is looking for a new active maintainer. To volunteer, just start hacking on your own repo, and I'll hand you the keys after I merge your first commit.
I have been traveling a lot these days, which is why I didn't notice this email before. But even after I read it, I got contradictory feelings.
On the one side, I strongly feel that something needs to be done with ASDF. As it is, it is an obstacle for the development and popularization of the implementation I work with. I keep getting reports of software that does not work with ASDF and ECL, libraries that render our standalone executable code useless because users are loading manually stuff from inside ASD files, because libraries are listed as load dependencies when they are only needed compile-time, because they are using ASDF symbols even in their own software without listing in any way a dependency for ASDF... Enough.
To this I have to confront several facts. One is that I am and will continue to be ECL's maintainer. The second one is that I like changing things and I like taking decisions and some of them are going to be very impopular (*). The third fact is that I have little time and this time will get even scarcer as I grow older and develop my own career in Physics.
What I mean with all this blah, blah, is that I would like to work on the next ASDF, with the help of those that volunteered to also collaborate in the release and bug control / reporting stuff, help which is really appreciated since I know how much a time saver that is. However, my volunteering comes with the expectation of no friction, no pressure and freedom to explore new ideas (some are quite clear in my head but need to see an implementation and a real life test). Do you think you can stand this? ;-)
The ASDF 2 source tree would remain as it is, with regular maintenance and extremely little and careful changes. Code could be reorganized so that ASDF 2 and 3 coexist sharing common bits (**), people would use either of them and I would mostly focus on the ASDF 3 part. This way if the project fails, we will always have ASDF 2, but at the same time hopefully things will improve and people will move towards the new, wonderful, powerful toolset that ASDF 3 should be.
Should people agree on my participation, I will definitely need help, for I know nothing of how ASDF's git tree or release process is organized.
Regards
Juanjo
(*) Forbid extending certain or even all ASDF's methods in user's systems (arbitrary load-op = no standalone or shipping), enforce test-op behavior, add install-op, add Makefile style rules to systems, merge load-op and load-source-op, split ASDF's source code, create a new ASDF loader that does not evaluate, introduce ASDF style warnings and loudly complain about crappy defsystems, crop the sources removing unused stuff (window's shortcuts?) and making it optional....
(**) I have a version of ASDF split into separate files that reconstruct the full picture. See http://tream.dreamhosters.com/git/index.php?p=lisp/asdf-decl.git&a=tree
Dear all,
I welcome both Vsevolod's and Juanjo's candidacies to overtake ASDF maintenance. Let the one who actually has time to do the hard work do the hard work and take over. However, while I generally agree with both of your plans with respect to the future of ASDF, I have a few remarks.
* Moving towards more rational release numbers is good. * A regular release cycle is only justified if there are regular changes. * Moving towards a fully declarative model is I think essential, though of course it does requires a clear split between backwards-compatible 2.x and incompatible 3.x. * supporting more ops is cool: test-op, install-op, etc., but of course is probably to be done as (standardized) extensions rather than as more stuff in asdf.lisp
Against a point that you both raised, I see no good reason whatsoever to split asdf in multiple files, though I can imagine plenty of bad reasons from past conversations, and I see plenty of good reasons not to do it.
Bad reasons to split: * ease of browsing? Nope, it's harder when split. You have to search for text in all files, lookup an index to see the order (that does matter). * speed of compiling? Nope, for hours of maintenance, you only get a few fractions of a second of speedup in the rare case that you only make a simple change to a file that leaves all APIs untouched. And only on a few implementations where you can somehow achieve separate compilation through ad-hoc non-portable scripts. Full build takes a fraction of a second MORE time, not less time, when you first have to concatenate all the subfiles, anyway, because in the end, the reason we put it all into asdf.lisp is because it was all necessary to bootstrap ASDF. * ease of maintenance? Nope, additional maintenance nightmare. * ease of distribution? Nope, it's easier to distribute one Lisp file than a directory of files. * ease of deployment? Nope, it has to be loaded all at once, and there's no portable way to concatenate FASLs. * ease of bug tracking? Nope, it's easier to md5sum or diff one file than to try to assert and maintain coherence between tens of files. * smaller size? Nope, you only increase the total overhead, and don't diminish the intrinsic complexity and size. * more sizeable chunks? Nope, my screenful of emacs buffer remains of the same size.
I'm sure we could figure out or imagine more reasons and non-reasons. But in the end, before you do such a sweeping change, can we have a rationale?
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Question: How many Intel/Microsoft executives does it take to screw in a light bulb?
Answer: None, they simply make darkness an industry standard.
On 16 September 2010 17:42, Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com wrote:
On Sat, Sep 11, 2010 at 7:41 PM, Faré fahree@gmail.com wrote:
I hope that 2.008 will be my last ASDF release. I may still make emergency bug fixes (if any is needed), or merge patches sent to me, but I don't intend to actively develop ASDF anymore
Thanks for all your work up to now, Fare.
And so, ASDF is looking for a new active maintainer. To volunteer, just start hacking on your own repo, and I'll hand you the keys after I merge your first commit.
I have been traveling a lot these days, which is why I didn't notice this email before. But even after I read it, I got contradictory feelings. On the one side, I strongly feel that something needs to be done with ASDF. As it is, it is an obstacle for the development and popularization of the implementation I work with. I keep getting reports of software that does not work with ASDF and ECL, libraries that render our standalone executable code useless because users are loading manually stuff from inside ASD files, because libraries are listed as load dependencies when they are only needed compile-time, because they are using ASDF symbols even in their own software without listing in any way a dependency for ASDF... Enough. To this I have to confront several facts. One is that I am and will continue to be ECL's maintainer. The second one is that I like changing things and I like taking decisions and some of them are going to be very impopular (*). The third fact is that I have little time and this time will get even scarcer as I grow older and develop my own career in Physics. What I mean with all this blah, blah, is that I would like to work on the next ASDF, with the help of those that volunteered to also collaborate in the release and bug control / reporting stuff, help which is really appreciated since I know how much a time saver that is. However, my volunteering comes with the expectation of no friction, no pressure and freedom to explore new ideas (some are quite clear in my head but need to see an implementation and a real life test). Do you think you can stand this? ;-) The ASDF 2 source tree would remain as it is, with regular maintenance and extremely little and careful changes. Code could be reorganized so that ASDF 2 and 3 coexist sharing common bits (**), people would use either of them and I would mostly focus on the ASDF 3 part. This way if the project fails, we will always have ASDF 2, but at the same time hopefully things will improve and people will move towards the new, wonderful, powerful toolset that ASDF 3 should be. Should people agree on my participation, I will definitely need help, for I know nothing of how ASDF's git tree or release process is organized. Regards Juanjo (*) Forbid extending certain or even all ASDF's methods in user's systems (arbitrary load-op = no standalone or shipping), enforce test-op behavior, add install-op, add Makefile style rules to systems, merge load-op and load-source-op, split ASDF's source code, create a new ASDF loader that does not evaluate, introduce ASDF style warnings and loudly complain about crappy defsystems, crop the sources removing unused stuff (window's shortcuts?) and making it optional.... (**) I have a version of ASDF split into separate files that reconstruct the full picture. See http://tream.dreamhosters.com/git/index.php?p=lisp/asdf-decl.git&a=tree -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://juanjose.garciaripoll.googlepages.com
There is a rationale for the splitting of asdf.lisp.
* Sorry, but the excuse of searching for code in a file is lame :-) Would that also apply to any thousand lines code library?
* The bootstrapping code can be different for a shipped asdf (one that comes with the implementation) and for the asdf that is loaded by users. This can be activated by an implementation by choosing whether to use defpackage.lisp or something else. Doing this with the monolithic asfd.lisp is a hell.
* The limitation of having a single asdf.lisp file is not insurmountable. It is currently due to a problem in ASDF definitions, which force a compile-load-compile-load-- sequence for the files, not allowing a compile-compile-compile... + load-load-load... The current model has the problem that as soon as new classes are loaded, the ASDF that is driving the build breaks down. I understand that this can be solved writing a better asdf.asd that includes serial :in-order-to (:compile-op ...) dependencies. Another less pleasant alternative would be to build ASDF in a separate bootstrapping package and then transferring the old database of modules to the new version (i.e. changing the current bootstrapping code).
* In the meantime asdf.lisp may be automatically created and distributed together with the rest. This imposes no maintenance burden and it can be actually used to ensure that asdf.lisp is only created with new releases, leaving the other files and directories for daring users / maintainers.
* I have been able to identify structure in the code and splitting it into files which are actually independent and which are connected by a yet flaky API (methods principally). This has been the criterion for splitting. The current asdf.lisp is a mess where it is extremely hard to identify all the methods defined for a class.
* In the split form methods and classes are logically grouped. It makes it easy to restrict oneself to good coding practices, such as enforcing an API for the different components to communicate among each other, and looking for strategies to optimize and trim down the size of those "modules". In other words, splitting may enforce defining a better API and isolation components such as traverse or the file configuration facility, making it easier to replace those components in the future.
* In splitting and reorganizing the code several modules have been found that are actually _not_ used. Windows shortcuts for instance.
* The split development model accommodates very easily new optional modules which may also be _included_ in a prepackaged ASDF -- easier configurability and extension for "vendors" or "distributors". In particular one may add asdf-ecl... asdf-sbcl... for the bits that are not portable (dumping images, merging files...)
Juanjo
Juanjo wrote:
There is a rationale for the splitting of asdf.lisp.
...
- In the meantime asdf.lisp may be automatically created and distributed
together with the rest. This imposes no maintenance burden and it can be actually used to ensure that asdf.lisp is only created with new releases, leaving the other files and directories for daring users / maintainers.
Excellent point, and one which I believe renders the discussion moot. This is a powerful technique for creating bootstrap systems.
- In splitting and reorganizing the code several modules have been found
that are actually _not_ used. Windows shortcuts for instance.
I worked hard to get those in there (they had been written and unused years before I found them). They worked shortly before the ASDF2 redesign, and I had hoped they survived it intact. IMO, this is a critical feature that should not be removed. Scanning hundreds of directories for *.asd is not a fast operation (yes, I have 100+ CL libs on my boxes). Putting all those directories in a search path is not easy -- without using symlinks or shortcuts, at which point you might as well simply link the asd files themselves... No need to reinvent the wheel, when every major OS provides an idiomatic solution.
While I do not use MS-broken-windows regularly, rumor has it that most people do. If symlinks are supported on POSIX (usually through TRUENAME), then we should try to support shortcuts on MS.
ASDF has a very liberal view of loading performance. A mechanism for caching system definitions may help; but the uncached performance should not be ignored. I hate starting clisp and waiting a minute or two for it to go looking for (require :something).
- Daniel
On Wed, Sep 22, 2010 at 4:00 PM, dherring@tentpost.com wrote:
Juanjo wrote:
- In splitting and reorganizing the code several modules have been found
that are actually _not_ used. Windows shortcuts for instance.
I worked hard to get those in there (they had been written and unused years before I found them). They worked shortly before the ASDF2 redesign, and I had hoped they survived it intact. IMO, this is a critical feature that should not be removed.
I am not advocating it to be removed. I just pointed out that in going through the code, cleaning it up and reorganizing it, unused features came to the surface. I was actually surprised to see that this code, which is Windows dependent, was compiled and included for _all_ platforms, but the associated functions were not used _anywhere_.
Juanjo
On 9/22/10 Sep 22 -5:26 AM, Juan Jose Garcia-Ripoll wrote:
There is a rationale for the splitting of asdf.lisp.
...
- The bootstrapping code can be different for a shipped asdf (one that
comes with the implementation) and for the asdf that is loaded by users. This can be activated by an implementation by choosing whether to use defpackage.lisp or something else. Doing this with the monolithic asfd.lisp is a hell.
Can you explain this? I don't really get it. Presumably any implementation that wants to distribute ASDF can add its own "after hooks" in its own private copy. I'm clearly missing something here...
...
I see some of your points, but I'd encourage you to carefully consider what I think is the core of Faré's argument: the cost/benefit tradeoff.
There is a very strictly finite number of hours that can go into ASDF development. This will eat a big chunk of them, and the benefit gained by the community seems small relative to spending those hours on adding new functionality.
Also this will incur a substantial cost in terms of testing hours, which are also strictly finite (although larger than the development hours supply).
Finally, this seems to make more work for implementation maintainers that wish to bundle ASDF, although perhaps that could be mitigated...
At the end of the day, of course, these are your hours. If you think it's important enough, there's nobody stopping you from forking the repo and doing this work. But as you've said yourself, you are primarily the ECL maintainer. Is this the location where you will get the most value from your development hours?
Best, r
On Wed, Sep 22, 2010 at 4:35 PM, Robert Goldman rpgoldman@sift.info wrote:
On 9/22/10 Sep 22 -5:26 AM, Juan Jose Garcia-Ripoll wrote:> * The bootstrapping code can be different for a shipped asdf (one that
comes with the implementation) and for the asdf that is loaded by users. This can be activated by an implementation by choosing whether to use defpackage.lisp or something else. Doing this with the monolithic asfd.lisp is a hell.
Can you explain this? I don't really get it. Presumably any implementation that wants to distribute ASDF can add its own "after hooks" in its own private copy. I'm clearly missing something here...
I am not only talking about hooks but about to-come features, such as dumping "executables", or creating monolithic fasl (a fasl made of multiple components). This can be done using separate files for each implementation, much like swank does, and requires implementation-dependent code which would be a nightmare to code via #+/#-. Same thing goes for the execution of commands via run-program or the like, or handling of configuration files -- to which implementations might wish to hook.
I see some of your points, but I'd encourage you to carefully consider what I think is the core of Faré's argument: the cost/benefit tradeoff.
Nobody has really shown me _any_ cost, except that of having to open one or multiple files. Maintainance of each feature (compile-op, load-op, traverse, shell operations, configuration files) must and has to become independent. Otherwise something is broken in the development.
Also this will incur a substantial cost in terms of testing hours, which
are also strictly finite (although larger than the development hours supply).
No. Why? I mean, the separate components are bundled into a single asdf.lisp and this process is automated (make -f GNUmakefile asdf.lisp). There is no new testing burden. I am just advocating for some common sense in the ASDF codebase, organizing it in a readable and separated way.
Robert, do you realize I already did the job for you? It is not is as if I was asking people here to do anything http://tream.dreamhosters.com/git/?p=lisp/asdf-decl.git&a=summary
Finally, this seems to make more work for implementation maintainers that wish to bundle ASDF, although perhaps that could be mitigated...
How? There is no burden. asdf.lisp will be built for them. If they _really_ want to add features, they can. I am thinking of ECL: I *do* want to have features built in ASDF.
At the end of the day, of course, these are your hours. If you think it's important enough, there's nobody stopping you from forking the repo and doing this work. But as you've said yourself, you are primarily the ECL maintainer. Is this the location where you will get the most value from your development hours?
Hmm? I do not get the point of the whole discussion. The splitting _has_ been done, as shown before. I do not have to spend any more hours doing so. However your question generally applies to the whole purpose of getting involve in ASDF maintenance.
Juanjo
Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com writes:
On Wed, Sep 22, 2010 at 4:35 PM, Robert Goldman rpgoldman@sift.info wrote:
On 9/22/10 Sep 22 -5:26 AM, Juan Jose Garcia-Ripoll wrote: > * The bootstrapping code can be different for a shipped asdf > (one that comes with the implementation) and for the asdf that > is loaded by users. This can be activated by an > implementation by choosing whether to use defpackage.lisp or > something else. Doing this with the monolithic asfd.lisp is a > hell. Can you explain this? I don't really get it. Presumably any implementation that wants to distribute ASDF can add its own "after hooks" in its own private copy. I'm clearly missing something here...I am not only talking about hooks but about to-come features, such as dumping "executables", or creating monolithic fasl (a fasl made of multiple components). This can be done using separate files for each implementation, much like swank does, and requires implementation-dependent code which would be a nightmare to code via #+/#-. Same thing goes for the execution of commands via run-program or the like, or handling of configuration files -- to which implementations might wish to hook.
Perhaps a tool like shar, but preserving the loadability of the lisp sources would be what you need to be able to work efficiently with the asdf.lisp source:
http://git.informatimago.com/viewgit/index.php?a=viewblob&p=public/bin&a...
asdf.lisp could still be the main source, under DCVS and distributed as a single file, but to hack on it you could easily split it out in its components:
mkdir asdf cd asdf clar ../asdf.lisp emacs compile-op.lisp asdf-ecl.lisp clar ../asdf.lisp *.lisp
No. Why? I mean, the separate components are bundled into a single asdf.lisp and this process is automated (make -f GNUmakefile asdf.lisp). There is no new testing burden. I am just advocating for some common sense in the ASDF codebase, organizing it in a readable and separated way.
This is an alternative, indeed.
asdf.lisp: $(ASDF_SOURCES) clar asdf.lisp $(ASDF_SOURCES)
On Wed, Sep 22, 2010 at 8:37 PM, Pascal J. Bourguignon < pjb@informatimago.com> wrote:
No. Why? I mean, the separate components are bundled into a single asdf.lisp and this process is automated (make -f GNUmakefile asdf.lisp). There is no new testing burden. I am just advocating for some common sense in the ASDF codebase, organizing it in a readable and separated way.
This is an alternative, indeed.
asdf.lisp: $(ASDF_SOURCES) clar asdf.lisp $(ASDF_SOURCES)
Indeed, eventually one should move my GNUmakefile line from using sed to using some Lisp implementation for repackaging asdf.lisp, ensuring a version number, etc.
Incidentally, I am still amazed at how people here find a 4k source file manageable at all. No organization, functions spread throught the file, no clear navigation structure... If I have a directory with 10 small files, each one devoted to a separate and clear task, I know where I have to go to edit things for that particular topic.
Juanjo
Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com writes:
On Wed, Sep 22, 2010 at 8:37 PM, Pascal J. Bourguignon pjb@informatimago.com wrote:
> No. Why? I mean, the separate components are bundled into a > single asdf.lisp and this process is automated (make -f > GNUmakefile asdf.lisp). There is no new testing burden. I am > just advocating for some common sense in the ASDF codebase, > organizing it in a readable and separated way. This is an alternative, indeed. asdf.lisp: $(ASDF_SOURCES) clar asdf.lisp $(ASDF_SOURCES)Indeed, eventually one should move my GNUmakefile line from using sed to using some Lisp implementation for repackaging asdf.lisp, ensuring a version number, etc.
Incidentally, I am still amazed at how people here find a 4k source file manageable at all. No organization, functions spread throught the file, no clear navigation structure... If I have a directory with 10 small files, each one devoted to a separate and clear task, I know where I have to go to edit things for that particular topic.
Some early C programmers even put one function per file. I guess it can be useful if you use vi...
But when you have a smart editor such as emacs, with at least a good search function, or even a find-tag feature, it doesn't really matter where your functions are stored. Actually, file management should be left up to the system and the programmer should not care about the storage of their functions. Like in Smalltalk for example. So instead of dealing with a linear sequence of functions, you can rather work on a tree or graph of functions, and navigate from one to the other directly.
That said, I too like to have things gathered up in a somewhat organized way, so that I may browse and revise the code in a meaningful way. But there are probably better ways to do it. In particular when you consider multiple dispatch in CLOS, you could have several ways to sort out the methods, depending on what you're doing. So it would be better to let the tools gather the code dynamically. (eg. get all the methods applying to objects of a given class, or get all the functions in the call graph rooted from a given function, things like that).
On Thu, Sep 23, 2010 at 4:27 AM, Pascal J. Bourguignon < pjb@informatimago.com> wrote:
But when you have a smart editor such as emacs, with at least a good search function, or even a find-tag feature, it doesn't really matter where your functions are stored.
I use emacs, and I am not thinking about searching functions, which has been also repeated to me privately as a reason (!) to have everything in one file, but rather in organizing the code and giving it a navigational structure. I find it useful at least for myself to split things into directories according to their role, group functions that belong together at least in my mind and work only on a small set of files at a time.
That said, there are times when this is impossible, as it is the problem with ECL, which has humongous files with hundreds of functions per-file. In this case there are two issues at stake, one is that C only makes optimal function calls to functions in the same file (maybe that changes with new link time optimizers) and the other one is to avoid exporting symbols.
Actually, file management should be
left up to the system and the programmer should not care about the storage of their functions. Like in Smalltalk for example. So instead of dealing with a linear sequence of functions, you can rather work on a tree or graph of functions, and navigate from one to the other directly.
Sorry, but I always found Smalltalk's way of working confusing. You just throw everything in a huge, deep, dark bag and everything is at your hand's reach, but that does not mean one gets a really organized project out of it. For me programs are more than just a collection of functions, but that is my personal taste.
That said, I too like to have things gathered up in a somewhat organized way, so that I may browse and revise the code in a meaningful way. But there are probably better ways to do it. In particular when you consider multiple dispatch in CLOS, you could have several ways to sort out the methods, depending on what you're doing.
My point is that even if there are many ways to do so, one of them imprints the actual view of the library or program organization. Using code layout as information for newcomers is fine and that is what I am trying to do -- being kind of a newcomer to lots of ASDF parts.
Juanjo
Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com writes:
On Thu, Sep 23, 2010 at 4:27 AM, Pascal J. Bourguignon pjb@informatimago.com wrote:
But when you have a smart editor such as emacs, with at least a good search function, or even a find-tag feature, it doesn't really matter where your functions are stored.I use emacs, and I am not thinking about searching functions, which has been also repeated to me privately as a reason (!) to have everything in one file, but rather in organizing the code and giving it a navigational structure. I find it useful at least for myself to split things into directories according to their role, group functions that belong together at least in my mind and work only on a small set of files at a time.
That said, there are times when this is impossible, as it is the problem with ECL, which has humongous files with hundreds of functions per-file. In this case there are two issues at stake, one is that C only makes optimal function calls to functions in the same file (maybe that changes with new link time optimizers) and the other one is to avoid exporting symbols.
In C, these problems can be solved with #include.
----(compile-unit.c)-------- #include <source1.c> #include <source2.c> #include <source3.c> // <- not .h, but .c ----------------------------
Actually, file management should be left up to the system and the programmer should not care about the storage of their functions. Like in Smalltalk for example. So instead of dealing with a linear sequence of functions, you can rather work on a tree or graph of functions, and navigate from one to the other directly.Sorry, but I always found Smalltalk's way of working confusing. You just throw everything in a huge, deep, dark bag and everything is at your hand's reach, but that does not mean one gets a really organized project out of it. For me programs are more than just a collection of functions, but that is my personal taste.
They provide a browser to organize things by category of classes and category of methods in addition to the class hiearachy.
That said, I too like to have things gathered up in a somewhat organized way, so that I may browse and revise the code in a meaningful way. But there are probably better ways to do it. In particular when you consider multiple dispatch in CLOS, you could have several ways to sort out the methods, depending on what you're doing.My point is that even if there are many ways to do so, one of them imprints the actual view of the library or program organization. Using code layout as information for newcomers is fine and that is what I am trying to do -- being kind of a newcomer to lots of ASDF parts.
Juanjo
Faré fahree@gmail.com writes:
Bad reasons to split:
- ease of browsing? Nope, it's harder when split.
You have to search for text in all files, lookup an index to see the order (that does matter).
- speed of compiling? Nope, for hours of maintenance, you only get a few
fractions of a second of speedup in the rare case that you only make a simple change to a file that leaves all APIs untouched. And only on a few implementations where you can somehow achieve separate compilation through ad-hoc non-portable scripts. Full build takes a fraction of a second MORE time, not less time, when you first have to concatenate all the subfiles, anyway, because in the end, the reason we put it all into asdf.lisp is because it was all necessary to bootstrap ASDF.
- ease of maintenance? Nope, additional maintenance nightmare.
- ease of distribution? Nope, it's easier to distribute one Lisp file
than a directory of files.
- ease of deployment? Nope, it has to be loaded all at once,
and there's no portable way to concatenate FASLs.
- ease of bug tracking? Nope, it's easier to md5sum or diff one file
than to try to assert and maintain coherence between tens of files.
- smaller size? Nope, you only increase the total overhead,
and don't diminish the intrinsic complexity and size.
- more sizeable chunks? Nope, my screenful of emacs buffer remains
of the same size.
I'm sure we could figure out or imagine more reasons and non-reasons. But in the end, before you do such a sweeping change, can we have a rationale?
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Question: How many Intel/Microsoft executives does it take to screw in a light bulb?
Answer: None, they simply make darkness an industry standard.
Let's agree to split when it reaches half the max emacs buffer size.
-
dherring＠tentpost.com -
Faré -
Juan Jose Garcia-Ripoll -
pjb＠informatimago.com -
Robert Goldman -
Vsevolod Dyomkin