Question: could we handle this problem with the new asdf contrib idea? Could we distribute an ASDF:RUN-SHELL-COMMAND based in the XCVB code base along with ASDF using a contrib system?
I am reluctant to see RUN-SHELL-COMMAND no longer be distributed with ASDF, since it seems to make a hash of our claims that ASDF can manage systems that mix other code with common lisp code. I don't believe that invoking external facilities should rely on additional ASDF system dependencies. But I am sympathetic to the idea that RUN-SHELL-COMMAND is too big to fit in asdf.lisp, if done properly.
A second question: can we provide something like autoload for RUN-SHELL-COMMAND, based on ASDF:LOAD-SYSTEM, to minimize dislocation? If we have something like autoload, then people won't have to rejigger their system structures if we deprecate R-S-C away. Indeed, if we give them a better version, then they would just be a step ahead.
Finally, if we provide a better R-S-C, how do users differentiate? #+asdf2 is simply too coarse a tool now. #+asdf-run-shell-command ?
Cheers, r
On Fri, Oct 21, 2011 at 16:06, Robert Goldman rpgoldman@sift.info wrote:
Question: could we handle this problem with the new asdf contrib idea? Could we distribute an ASDF:RUN-SHELL-COMMAND based in the XCVB code base along with ASDF using a contrib system?
Probably. Or it could remain packaged separately.
I think it's a bad idea to duplicate code between XCVB and ASDF-CONTRIB. XCVB just *needs* that code in its driver and cannot import it: the XCVB driver, like ASDF, has this "everything in one file" restriction. XCVB might as well export this functionality.
Maybe I should detach xcvb-driver from the rest of xcvb, or define a package that just imports and exports select parts of XCVB-DRIVER and insulates users from change there, possibly adding value not strictly necessary in the driver. We'll see.
With Quicklisp, it shouldn't matter where exactly the code is coming from, as long as it is good quality software.
I am reluctant to see RUN-SHELL-COMMAND no longer be distributed with ASDF, since it seems to make a hash of our claims that ASDF can manage systems that mix other code with common lisp code.
ASDF cannot manage such code out of the box. It needs extensions, anyway. Deprecating R-S-C is just a way to acknowledge that it is the WRONG tool to write robust portable such extensions, and that better tools are available.
I don't believe that invoking external facilities should rely on additional ASDF system dependencies. But I am sympathetic to the idea that RUN-SHELL-COMMAND is too big to fit in asdf.lisp, if done properly.
Why can't ASDF extensions rely on basic CL libraries? The very point of making ASDF itself minimal yet extensible is to bootstrap functionality in such libraries and extensions.
A second question: can we provide something like autoload for RUN-SHELL-COMMAND, based on ASDF:LOAD-SYSTEM, to minimize dislocation?
That's actually quite a neat idea! Whenever I gut out R-S-C (not until known users have been fixed), I'll leave this instead:
(defun run-shell-command (fmt &rest args) "This function is badly designed and deprecated. This thin wrapper provides a compatibility layer for legacy code. Please do not use in your code; instead, directly use xcvb-driver or some other well-designed, robustly implemented, ported and supported interface" (asdf:load-system :xcvb-driver) (funcall (find-symbol (string :run-program/for-side-effects) :xcvb-driver) (apply 'format nil fmt args)))
And maybe a define-compiler-macro to emit a style-warning at compile-time?
If we have something like autoload, then people won't have to rejigger their system structures if we deprecate R-S-C away. Indeed, if we give them a better version, then they would just be a step ahead.
That's a great suggestion.
Finally, if we provide a better R-S-C, how do users differentiate? #+asdf2 is simply too coarse a tool now. #+asdf-run-shell-command ?
One more reason to direct users to a better interface. Using format will never provide a robust interface to escaping characters for the shell, much less portably between Windows and Unix. Leaky abstractions SUCK. R-S-C has wet its pants and made a puddle already.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
On 10/21/11 Oct 21 -3:49 PM, Faré wrote:
On Fri, Oct 21, 2011 at 16:06, Robert Goldman rpgoldman@sift.info wrote:
Question: could we handle this problem with the new asdf contrib idea? Could we distribute an ASDF:RUN-SHELL-COMMAND based in the XCVB code base along with ASDF using a contrib system?
Probably. Or it could remain packaged separately.
I think it's a bad idea to duplicate code between XCVB and ASDF-CONTRIB. XCVB just *needs* that code in its driver and cannot import it: the XCVB driver, like ASDF, has this "everything in one file" restriction. XCVB might as well export this functionality.
Maybe I should detach xcvb-driver from the rest of xcvb, or define a package that just imports and exports select parts of XCVB-DRIVER and insulates users from change there, possibly adding value not strictly necessary in the driver. We'll see.
With Quicklisp, it shouldn't matter where exactly the code is coming from, as long as it is good quality software.
I am reluctant to see RUN-SHELL-COMMAND no longer be distributed with ASDF, since it seems to make a hash of our claims that ASDF can manage systems that mix other code with common lisp code.
ASDF cannot manage such code out of the box. It needs extensions, anyway. Deprecating R-S-C is just a way to acknowledge that it is the WRONG tool to write robust portable such extensions, and that better tools are available.
I don't agree with the claim "better tools are available" yet. ASDF is the standard for the community. Making people use a different tool is like insisting that C programmers use something other than `make`. It raises the bar too high.
I don't believe that invoking external facilities should rely on additional ASDF system dependencies. But I am sympathetic to the idea that RUN-SHELL-COMMAND is too big to fit in asdf.lisp, if done properly.
Why can't ASDF extensions rely on basic CL libraries? The very point of making ASDF itself minimal yet extensible is to bootstrap functionality in such libraries and extensions.
I don't agree because the cost of adding these libraries is too high. To do this now for a system I want to rely on I must (1) locate the new run-shell-command library (2) blow that run-shell-command library into my revision control system (3) forevermore monitor the library for modifications.
[For many applications quicklisp may help, but for keeping our commercial stuff up and running, we need absolute control, and that means keeping stuff under revision control. I believe that just about any company developing software would be in the same position.]
This is especially true since the CL community does not yet have a good solution to (3).
Currently, the implementations attempt to keep good working versions of ASDF out there. There is no corresponding crutch for arbitrary libraries (which would include our hypothetical RUN-SHELL-COMMAND).
For software that does not intend to be fully portable, for one or two calls to RUN-SHELL-COMMAND, it is far easier to do your own #+allegro, #+sbcl, #+ccl (or whatever) than it is to import and track a new library. For one thing, I am confident that if I get the small number of individual calls right, they will be truly right and will stay right.
In fact, in the original two applications that got me started finding problems with ASDF:RUN-SHELL-COMMAND, I did just that. They will probably only ever run on ACL, SBCL and maybe CCL. The conditional compilation alternatives seem far less rickety than trying to get RUN-SHELL-COMMAND to work properly, or to grok another portability library.
Of course, an end application case, such as I have in mind is a different use case from a core library such as, e.g., CFFI, where you want full portability, and the advantages of trusting someone else to get the shell commands right on all platforms pay off better.
Maybe I'm starting to agree with you --- ASDF:RUN-SHELL-COMMAND will never be good enough, and programmers are just better off avoiding it. Better not to have anything than to have something that does The Wrong Thing.
Cheers, r
On Fri, Oct 21, 2011 at 17:11, Robert Goldman rpgoldman@sift.info wrote:
I don't agree with the claim "better tools are available" yet. ASDF is the standard for the community. Making people use a different tool is like insisting that C programmers use something other than `make`. It raises the bar too high.
I didn't mean "ASDF is deprecated", just "ASDF:RUN-SHELL-COMMAND is deprecated".
ASDF itself doesn't use RUN-SHELL-COMMAND. R-S-C has a bad interface, a fragile implementation, and can't be fixed. Other alternatives exist that are easier to use in all cases; XCVB-DRIVER:RUN-PROGRAM/FOR-SIDE-EFFECTS is an example. It will take either a shell command string (but then, it offers you functions to properly escape the contents) or a list of program and arguments (that will be interpreted according to your platform's C API).
There isn't EVER any good reason to use R-S-C. By definition, if you use it, you're using ASDF, and could thus defsystem-depends-on a better library using ASDF. If you don't care about portability, then you can already use your platform's interface, that is much more powerful than RUN-SHELL-COMMAND. If you do care about portability, you shouldn't be using R-S-C. Finally, format controls are a *very* fragile way to produce shell commands in presence of special characters in file names or directory names (think Windows). R-S-C encourages sloppy code.
Die, RUN-SHELL-COMMAND, die!
(And yes, I'll be supporting it until it's dead and buried; that's what I've been doing with ASDF.)
I don't agree because the cost of adding these libraries is too high. To do this now for a system I want to rely on I must (1) locate the new run-shell-command library (2) blow that run-shell-command library into my revision control system (3) forevermore monitor the library for modifications.
Quicklisp makes this cost much lower. And if you Lisp infrastructure doesn't allow you to easily maintain libraries, you have a problem of your own.
[For many applications quicklisp may help, but for keeping our commercial stuff up and running, we need absolute control, and that means keeping stuff under revision control. I believe that just about any company developing software would be in the same position.]
Yes, at Google, we have plenty of external libraries under revision control. I'm using git to manage the branching between our code and upstream.
This is especially true since the CL community does not yet have a good solution to (3).
That's not specific to the CL community, is it? I remember experiencing just as much DLL hell with C/C++ code. One solution that works: git modules. But once again, any problem with distribution of Lisp libraries has to be solved outside ASDF itself. Quicklisp and clbuild are attempts to help.
Currently, the implementations attempt to keep good working versions of ASDF out there. There is no corresponding crutch for arbitrary libraries (which would include our hypothetical RUN-SHELL-COMMAND).
Neither should they be. We need implementations to package ASDF (and possibly a quicklisp installer?) because it's so basic a need, required to bootstrap more things. Once you can bootstrap, you don't need more.
For software that does not intend to be fully portable, for one or two calls to RUN-SHELL-COMMAND, it is far easier to do your own #+allegro, #+sbcl, #+ccl (or whatever) than it is to import and track a new library. For one thing, I am confident that if I get the small number of individual calls right, they will be truly right and will stay right.
But RUN-SHELL-COMMAND really invites you to write sloppy code that will bite you in the ass if it's deployed beyond your own machine.
In fact, in the original two applications that got me started finding problems with ASDF:RUN-SHELL-COMMAND, I did just that. They will probably only ever run on ACL, SBCL and maybe CCL. The conditional compilation alternatives seem far less rickety than trying to get RUN-SHELL-COMMAND to work properly, or to grok another portability library.
I've started thinking of XVCB-DRIVER as just as basic as ASDF. (asdf:load-system :xcvb-driver)
Of course, an end application case, such as I have in mind is a different use case from a core library such as, e.g., CFFI, where you want full portability, and the advantages of trusting someone else to get the shell commands right on all platforms pay off better.
Certainly. Yet even then, R-S-C is only inviting trouble.
I agree that there's an issue with getting all the CL batteries included everywhere we need them.
Maybe I'm starting to agree with you --- ASDF:RUN-SHELL-COMMAND will never be good enough, and programmers are just better off avoiding it. Better not to have anything than to have something that does The Wrong Thing.
Well at least the various known R-S-C bugs seem to be fixed on Unix. But R-S-C will never work well on Windows, and never be full-featured wrt output redirection.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
-
Faré -
Robert Goldman