On 9/12/10 Sep 12 -11:48 AM, Vsevolod Dyomkin wrote:

Hi,

Some random comments....

I have spent some time this year familiarizing myself with ASDF, and afterwards I think, that it's a great build-tool (possibly, the best one around, actually), which has a lot of hidden potential. So I'd be willing to help as the maintainer or one of the maintainers.

....

As a Release manager I'd do the following: - establish a regular release cycle (bi-monthly), transition to follow the Rational version policy

Three points with respect to this: 1. We should probably have a definition of "rational version policy," at least by citation. 2. If the rational version policy is the numbering scheme I found by googling, I don't believe it is compatible with asdf:version-satisfies. I would suggest we avoid adopting a policy that doesn't meet that constraint. 3. Is this bi-monthly release policy wrt ASDF 3? In my fondest dreams ASDF2 is getting stable and the ASDF 2 release cycle should be "very rarely, never in the limit." I say this not just because I am looking forward to ASDF 2 being done, but also because ASDF is critically foundational to the entire open source CL community. In order for members of this community to be able to share work, ASDF's API needs to be very, very stable (see the paper Faré and I wrote about ASDF 2 development for a more thorough presentation of the need for stability).

- finish creating a comprehensive test-suite, and organize some automatic testing process

This would be very good. BTW, I have just discovered that there are at least three different ways to start CCL, and only one of them is currently exercised by our test suite.

- move development to github, where there is some rather convenient infrastructure, like issue tracking (leaving a mirror on common-lisp.net <http://common-lisp.net>)

Is there some reason that common-lisp.net + launchpad is unsatisfactory?

- work on improving documentation (also I'm currently doing a series of articles about ASDF in Russian in my blog http://lisp-univ-etc.blogspot.com, that I'm also going to translate to English eventually) - continue work on separating ASDF itself and some support subsystems - establish some basic contribution guidelines - answer questions in the mailing list - contribute to bug-fixing

This sounds great. Best, Robert

On 9/12/10 Sep 12 -1:55 PM, Vsevolod Dyomkin wrote:

On Sun, Sep 12, 2010 at 8:09 PM, Robert Goldman <rpgoldman@sift.info <mailto:rpgoldman@sift.info>> wrote:

On 9/12/10 Sep 12 -11:48 AM, Vsevolod Dyomkin wrote: > Hi,

Some random comments....

....

2. If the rational version policy is the numbering scheme I found by googling, I don't believe it is compatible with asdf:version-satisfies. I would suggest we avoid adopting a policy that doesn't meet that constraint.

Last time I looked, version-satisfies quite-happily handled exact versions, like "1.2.3", which also comply with the "rational" policy. At the same time, current versions, like 2.103 are actually 2.1.3 in disguise, which is a little annoying. Yet, the biggest benefit of the rational policy (or at least some policy) is that it's predictable.

I don't believe that is correct. Here is a line I have ripped from version-satisfies-p, which seems to clearly show that 2.103 is (2 103) not (2 1 3): ASDF(76): (mapcar #'parse-integer (split-string "2.103" :separator ".")) (2 103)

3. Is this bi-monthly release policy wrt ASDF 3? In my fondest dreams ASDF2 is getting stable and the ASDF 2 release cycle should be "very rarely, never in the limit." I say this not just because I am looking forward to ASDF 2 being done, but also because ASDF is critically foundational to the entire open source CL community. In order for members of this community to be able to share work, ASDF's API needs to be very, very stable (see the paper Faré and I wrote about ASDF 2 development for a more thorough presentation of the need for stability).

Rarely a release should break backwards compatibility. But from what I see now, there's quite a lot of fixes to ASDF (at least a couple each month), and I don't forsee the change to such situation.

I certainly hope you are wrong in thinking this won't change, but fine. I think we are still going to want a release policy for ASDF 3 and ASDF 2 separately, with the ASDF API being stable over ASDF 2 releases.

> - finish creating a comprehensive test-suite, and organize some > automatic testing process

This would be very good. BTW, I have just discovered that there are at least three different ways to start CCL, and only one of them is currently exercised by our test suite.

> - move development to github, where there is some rather > convenient infrastructure, like issue tracking (leaving a mirror on > common-lisp.net <http://common-lisp.net> <http://common-lisp.net>)

Is there some reason that common-lisp.net <http://common-lisp.net> + launchpad is unsatisfactory?

It's much less integrated and more clumsy (I filed several bugs through launchpad myself and don't like the experience). It seems, that github is a much friendlier place for that, and it is developing pretty fast. Yet, I don't think, that it is critical, so if other contributors would like to stay with the current variant, I don't object.

I am just feeling the need for some stability, having been through CVS and git already (and the git transition was pretty bumpy). Getting all the information OUT of launchpad and into a different issue tracker seems painful. However, if someone wants to take the trouble of actually porting all the launchpad data, who am I to object? Best, r

...

ASDF(76): (mapcar #'parse-integer                   (split-string "2.103" :separator ".")) (2 103)

And that is what's annoying, since actually (semantically) 103 is meant as 1.3, no?

Yup, and it's all my fault for failing to switch numbering during the 2.0 release. Hopefully, 3.0 will fix that. BTW, I just committed 2.129 to fix one more bug. Sigh. I hope a new maintainer will release that. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Socialists, who want everyone enslaved, don't object to private slave-ownership that it implies slaves, but that it implies non-slaves.

Dear all, I welcome both Vsevolod's and Juanjo's candidacies to overtake ASDF maintenance. Let the one who actually has time to do the hard work do the hard work and take over. However, while I generally agree with both of your plans with respect to the future of ASDF, I have a few remarks. * Moving towards more rational release numbers is good. * A regular release cycle is only justified if there are regular changes. * Moving towards a fully declarative model is I think essential, though of course it does requires a clear split between backwards-compatible 2.x and incompatible 3.x. * supporting more ops is cool: test-op, install-op, etc., but of course is probably to be done as (standardized) extensions rather than as more stuff in asdf.lisp Against a point that you both raised, I see no good reason whatsoever to split asdf in multiple files, though I can imagine plenty of bad reasons from past conversations, and I see plenty of good reasons not to do it. Bad reasons to split: * ease of browsing? Nope, it's harder when split. You have to search for text in all files, lookup an index to see the order (that does matter). * speed of compiling? Nope, for hours of maintenance, you only get a few fractions of a second of speedup in the rare case that you only make a simple change to a file that leaves all APIs untouched. And only on a few implementations where you can somehow achieve separate compilation through ad-hoc non-portable scripts. Full build takes a fraction of a second MORE time, not less time, when you first have to concatenate all the subfiles, anyway, because in the end, the reason we put it all into asdf.lisp is because it was all necessary to bootstrap ASDF. * ease of maintenance? Nope, additional maintenance nightmare. * ease of distribution? Nope, it's easier to distribute one Lisp file than a directory of files. * ease of deployment? Nope, it has to be loaded all at once, and there's no portable way to concatenate FASLs. * ease of bug tracking? Nope, it's easier to md5sum or diff one file than to try to assert and maintain coherence between tens of files. * smaller size? Nope, you only increase the total overhead, and don't diminish the intrinsic complexity and size. * more sizeable chunks? Nope, my screenful of emacs buffer remains of the same size. I'm sure we could figure out or imagine more reasons and non-reasons. But in the end, before you do such a sweeping change, can we have a rationale? [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Question: How many Intel/Microsoft executives does it take to screw in a light bulb? Answer: None, they simply make darkness an industry standard. On 16 September 2010 17:42, Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com> wrote:

On Sat, Sep 11, 2010 at 7:41 PM, Faré <fahree@gmail.com> wrote:

...

I hope that 2.008 will be my last ASDF release. I may still make emergency bug fixes (if any is needed), or merge patches sent to me, but I don't intend to actively develop ASDF anymore

Thanks for all your work up to now, Fare.

...

And so, ASDF is looking for a new active maintainer. To volunteer, just start hacking on your own repo, and I'll hand you the keys after I merge your first commit.

I have been traveling a lot these days, which is why I didn't notice this email before. But even after I read it, I got contradictory feelings. On the one side, I strongly feel that something needs to be done with ASDF. As it is, it is an obstacle for the development and popularization of the implementation I work with. I keep getting reports of software that does not work with ASDF and ECL, libraries that render our standalone executable code useless because users are loading manually stuff from inside ASD files, because libraries are listed as load dependencies when they are only needed compile-time, because they are using ASDF symbols even in their own software without listing in any way a dependency for ASDF... Enough. To this I have to confront several facts. One is that I am and will continue to be ECL's maintainer. The second one is that I like changing things and I like taking decisions and some of them are going to be very impopular (*). The third fact is that I have little time and this time will get even scarcer as I grow older and develop my own career in Physics. What I mean with all this blah, blah, is that I would like to work on the next ASDF, with the help of those that volunteered to also collaborate in the release and bug control / reporting stuff, help which is really appreciated since I know how much a time saver that is. However, my volunteering comes with the expectation of no friction, no pressure and freedom to explore new ideas (some are quite clear in my head but need to see an implementation and a real life test). Do you think you can stand this? ;-) The ASDF 2 source tree would remain as it is, with regular maintenance and extremely little and careful changes. Code could be reorganized so that ASDF 2 and 3 coexist sharing common bits (**), people would use either of them and I would mostly focus on the ASDF 3 part. This way if the project fails, we will always have ASDF 2, but at the same time hopefully things will improve and people will move towards the new, wonderful, powerful toolset that ASDF 3 should be. Should people agree on my participation, I will definitely need help, for I know nothing of how ASDF's git tree or release process is organized. Regards Juanjo (*)  Forbid extending certain or even all ASDF's methods in user's systems (arbitrary load-op = no standalone or shipping), enforce test-op behavior, add install-op, add Makefile style rules to systems, merge load-op and load-source-op, split ASDF's source code, create a new ASDF loader that does not evaluate, introduce ASDF style warnings and loudly complain about crappy defsystems, crop the sources removing unused stuff (window's shortcuts?) and making it optional.... (**) I have a version of ASDF split into separate files that reconstruct the full picture. See http://tream.dreamhosters.com/git/index.php?p=lisp/asdf-decl.git&a=tree -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://juanjose.garciaripoll.googlepages.com

Juanjo wrote:

There is a rationale for the splitting of asdf.lisp. ... * In the meantime asdf.lisp may be automatically created and distributed together with the rest. This imposes no maintenance burden and it can be actually used to ensure that asdf.lisp is only created with new releases, leaving the other files and directories for daring users / maintainers.

Excellent point, and one which I believe renders the discussion moot. This is a powerful technique for creating bootstrap systems.

* In splitting and reorganizing the code several modules have been found that are actually _not_ used. Windows shortcuts for instance.

I worked hard to get those in there (they had been written and unused years before I found them). They worked shortly before the ASDF2 redesign, and I had hoped they survived it intact. IMO, this is a critical feature that should not be removed. Scanning hundreds of directories for *.asd is not a fast operation (yes, I have 100+ CL libs on my boxes). Putting all those directories in a search path is not easy -- without using symlinks or shortcuts, at which point you might as well simply link the asd files themselves... No need to reinvent the wheel, when every major OS provides an idiomatic solution. While I do not use MS-broken-windows regularly, rumor has it that most people do. If symlinks are supported on POSIX (usually through TRUENAME), then we should try to support shortcuts on MS. ASDF has a very liberal view of loading performance. A mechanism for caching system definitions may help; but the uncached performance should not be ignored. I hate starting clisp and waiting a minute or two for it to go looking for (require :something). - Daniel

On 9/22/10 Sep 22 -5:26 AM, Juan Jose Garcia-Ripoll wrote:

There is a rationale for the splitting of asdf.lisp.

...

* The bootstrapping code can be different for a shipped asdf (one that comes with the implementation) and for the asdf that is loaded by users. This can be activated by an implementation by choosing whether to use defpackage.lisp or something else. Doing this with the monolithic asfd.lisp is a hell.

Can you explain this? I don't really get it. Presumably any implementation that wants to distribute ASDF can add its own "after hooks" in its own private copy. I'm clearly missing something here... ... I see some of your points, but I'd encourage you to carefully consider what I think is the core of Faré's argument: the cost/benefit tradeoff. There is a very strictly finite number of hours that can go into ASDF development. This will eat a big chunk of them, and the benefit gained by the community seems small relative to spending those hours on adding new functionality. Also this will incur a substantial cost in terms of testing hours, which are also strictly finite (although larger than the development hours supply). Finally, this seems to make more work for implementation maintainers that wish to bundle ASDF, although perhaps that could be mitigated... At the end of the day, of course, these are your hours. If you think it's important enough, there's nobody stopping you from forking the repo and doing this work. But as you've said yourself, you are primarily the ECL maintainer. Is this the location where you will get the most value from your development hours? Best, r

Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com> writes:

On Wed, Sep 22, 2010 at 4:35 PM, Robert Goldman <rpgoldman@sift.info> wrote:

On 9/22/10 Sep 22 -5:26 AM, Juan Jose Garcia-Ripoll wrote: > * The bootstrapping code can be different for a shipped asdf > (one that comes with the implementation) and for the asdf that > is loaded by users. This can be activated by an > implementation by choosing whether to use defpackage.lisp or > something else. Doing this with the monolithic asfd.lisp is a > hell.

Can you explain this?  I don't really get it.  Presumably any implementation that wants to distribute ASDF can add its own "after hooks" in its own private copy.  I'm clearly missing something here...

I am not only talking about hooks but about to-come features, such as dumping "executables", or creating monolithic fasl (a fasl made of multiple components). This can be done using separate files for each implementation, much like swank does, and requires implementation-dependent code which would be a nightmare to code via #+/#-. Same thing goes for the execution of commands via run-program or the like, or handling of configuration files -- to which implementations might wish to hook.

Perhaps a tool like shar, but preserving the loadability of the lisp sources would be what you need to be able to work efficiently with the asdf.lisp source: http://git.informatimago.com/viewgit/index.php?a=viewblob&p=public/bin&h=f9d1d846ec9127fdbcd4c86837d66f827d2eb72d&hb=ce800092f0c3e22ec5af4f95b2c2079b3bebb0e4&f=clar asdf.lisp could still be the main source, under DCVS and distributed as a single file, but to hack on it you could easily split it out in its components: mkdir asdf cd asdf clar ../asdf.lisp emacs compile-op.lisp asdf-ecl.lisp clar ../asdf.lisp *.lisp

No. Why? I mean, the separate components are bundled into a single asdf.lisp and this process is automated (make -f GNUmakefile asdf.lisp). There is no new testing burden. I am just advocating for some common sense in the ASDF codebase, organizing it in a readable and separated way.

This is an alternative, indeed. asdf.lisp: $(ASDF_SOURCES) clar asdf.lisp $(ASDF_SOURCES) -- __Pascal Bourguignon__ http://www.informatimago.com/

Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com> writes:

On Wed, Sep 22, 2010 at 8:37 PM, Pascal J. Bourguignon <pjb@informatimago.com> wrote:

> No. Why? I mean, the separate components are bundled into a > single asdf.lisp and this process is automated (make -f > GNUmakefile asdf.lisp). There is no new testing burden. I am > just advocating for some common sense in the ASDF codebase, > organizing it in a readable and separated way.

This is an alternative, indeed.

asdf.lisp: $(ASDF_SOURCES)        clar asdf.lisp $(ASDF_SOURCES)

Indeed, eventually one should move my GNUmakefile line from using sed to using some Lisp implementation for repackaging asdf.lisp, ensuring a version number, etc.

Incidentally, I am still amazed at how people here find a 4k source file manageable at all. No organization, functions spread throught the file, no clear navigation structure... If I have a directory with 10 small files, each one devoted to a separate and clear task, I know where I have to go to edit things for that particular topic.

Some early C programmers even put one function per file. I guess it can be useful if you use vi... But when you have a smart editor such as emacs, with at least a good search function, or even a find-tag feature, it doesn't really matter where your functions are stored. Actually, file management should be left up to the system and the programmer should not care about the storage of their functions. Like in Smalltalk for example. So instead of dealing with a linear sequence of functions, you can rather work on a tree or graph of functions, and navigate from one to the other directly. That said, I too like to have things gathered up in a somewhat organized way, so that I may browse and revise the code in a meaningful way. But there are probably better ways to do it. In particular when you consider multiple dispatch in CLOS, you could have several ways to sort out the methods, depending on what you're doing. So it would be better to let the tools gather the code dynamically. (eg. get all the methods applying to objects of a given class, or get all the functions in the call graph rooted from a given function, things like that). -- __Pascal Bourguignon__ http://www.informatimago.com/

Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com> writes:

On Thu, Sep 23, 2010 at 4:27 AM, Pascal J. Bourguignon <pjb@informatimago.com> wrote:

But when you have a smart editor such as emacs, with at least a good search function, or even a find-tag feature, it doesn't really matter where your functions are stored.

I use emacs, and I am not thinking about searching functions, which has been also repeated to me privately as a reason (!) to have everything in one file, but rather in organizing the code and giving it a navigational structure. I find it useful at least for myself to split things into directories according to their role, group functions that belong together at least in my mind and work only on a small set of files at a time.

That said, there are times when this is impossible, as it is the problem with ECL, which has humongous files with hundreds of functions per-file. In this case there are two issues at stake, one is that C only makes optimal function calls to functions in the same file (maybe that changes with new link time optimizers) and the other one is to avoid exporting symbols.

In C, these problems can be solved with #include. ----(compile-unit.c)-------- #include <source1.c> #include <source2.c> #include <source3.c> // <- not .h, but .c ----------------------------

Actually, file management should be left up to the system and the programmer should not care about the storage of their functions.  Like in Smalltalk for example.  So instead of dealing with a linear sequence of functions, you can rather work on a tree or graph of functions, and navigate from one to the other directly.

Sorry, but I always found Smalltalk's way of working confusing. You just throw everything in a huge, deep, dark bag and everything is at your hand's reach, but that does not mean one gets a really organized project out of it. For me programs are more than just a collection of functions, but that is my personal taste.

They provide a browser to organize things by category of classes and category of methods in addition to the class hiearachy.

That said, I too like to have things gathered up in a somewhat organized way, so that I may browse and revise the code in a meaningful way.  But there are probably better ways to do it.  In particular when you consider multiple dispatch in CLOS, you could have several ways to sort out the methods, depending on what you're doing.

My point is that even if there are many ways to do so, one of them imprints the actual view of the library or program organization. Using code layout as information for newcomers is fine and that is what I am trying to do -- being kind of a newcomer to lots of ASDF parts.

Juanjo

-- __Pascal Bourguignon__ http://www.informatimago.com/