In a potentially memory constrained environment, I need to portably (across sbcl and ccl at least) process a potentially large (multiple GiB) stream of bytes output from a Linux command. For the curious, "process" here means encrypt with a block cipher and push to the network; whereas the UNIX command is the output of "btrfs send".
I wanted to confirm with ASDF developers that as far as I can tell from wrangling with UIOP:RUN-PROGRAM, it isn't going to do what I want because there is no "asynchronous" mode. UIOP:RUN-PROGRAM cannot be "run in the background" like SBCL's SB-EXT:RUN-PROGRAM with :wait nil that allows the output of the sub-process to be gathered and processed in blocks. So, it would always be the case that UIOP/RUN-PROGRAM::SLURP-INPUT-STREAM has "gone through" all the (potentially huge) output of the sub-process before UIOP:RUN-PROGRAM can return.
Note that I'm not complaining about UIOP:RUN-PROGAM: it certainly bridges a lot of implementation specific details of the use case for "Execute this command; let me process its output". I'm just trying to affirm my understanding of UIOP and (regrettable) need to move off to do my own CCL/SBCL implementation for my needs.
If my understanding is correct, the UIOP maintainers may wish to emphasize the synchronous nature of UIOP:RUN-PROGRAM in its docstring.
In a potentially memory constrained environment, I need to portably (across sbcl and ccl at least) process a potentially large (multiple GiB) stream of bytes output from a Linux command. For the curious, "process" here means encrypt with a block cipher and push to the network; whereas the UNIX command is the output of "btrfs send".
I wanted to confirm with ASDF developers that as far as I can tell from wrangling with UIOP:RUN-PROGRAM, it isn't going to do what I want because there is no "asynchronous" mode. UIOP:RUN-PROGRAM cannot be "run in the background" like SBCL's SB-EXT:RUN-PROGRAM with :wait nil that allows the output of the sub-process to be gathered and processed in blocks. So, it would always be the case that UIOP/RUN-PROGRAM::SLURP-INPUT-STREAM has "gone through" all the (potentially huge) output of the sub-process before UIOP:RUN-PROGRAM can return.
Yes, UIOP:RUN-PROGRAM is synchronous. For an asynchronous and portable function I suggest IOLIB/OS:CREATE-PROCESS.
On Tue, Sep 29, 2015 at 11:03 AM, Mark Evenson evenson@panix.com wrote:
I wanted to confirm with ASDF developers that as far as I can tell from wrangling with UIOP:RUN-PROGRAM, it isn't going to do what I want because there is no "asynchronous" mode.
UIOP:RUN-PROGRAM itself is synchronous only, but its internal UIOP::%RUN-PROGRAM is asynchronous on platforms that support it... which does not include ABCL at this time, but does include CCL and SBCL. The interface is under-documented, and which exact subset of features it supports depends on the underlying implementation; but obviously it works well enough to portably implement the synchronous UIOP:RUN-PROGRAM.
If my understanding is correct, the UIOP maintainers may wish to emphasize the synchronous nature of UIOP:RUN-PROGRAM in its docstring.
I'll update the docstrings.
If you only care about CCL and SBCL, you can use UIOP::%RUN-PROGRAM.
If you want a portable solution that requires an external library, try IOLIB/OS:CREATE-PROCESS.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The liberties of the American people [are] dependent upon the ballot-box, the jury-box, and the cartridge-box. — Frederick Douglass
On 2015/9/29 17:34, Faré wrote:
On Tue, Sep 29, 2015 at 11:03 AM, Mark Evenson evenson@panix.com wrote:
I wanted to confirm with ASDF developers that as far as I can tell from wrangling with UIOP:RUN-PROGRAM, it isn't going to do what I want because there is no "asynchronous" mode.
UIOP:RUN-PROGRAM itself is synchronous only, but its internal UIOP::%RUN-PROGRAM is asynchronous on platforms that support it... which does not include ABCL at this time, but does include CCL and SBCL. The interface is under-documented, and which exact subset of features it supports depends on the underlying implementation; but obviously it works well enough to portably implement the synchronous UIOP:RUN-PROGRAM.
Is their any informal ASDF policy on the stability unexported symbols like UIOP::%RUN-PROGRAM? I suspect y'all reserve the right to yank the rug at any point, right? Given that UIOP::%RUN-PROGRAM abstracts the Lisp implementations capable of asynchronous operation, I guess it is a little more stable.
[…]
If you only care about CCL and SBCL, you can use UIOP::%RUN-PROGRAM.
If you want a portable solution that requires an external library, try IOLIB/OS:CREATE-PROCESS.
The "external library" library part of IOLIB always tends to make me shy away.
@stellian and @fare: thanks for the advice.
On Tue, Sep 29, 2015 at 12:14 PM, Mark Evenson evenson@panix.com wrote:
Is their any informal ASDF policy on the stability unexported symbols like UIOP::%RUN-PROGRAM? I suspect y'all reserve the right to yank the rug at any point, right? Given that UIOP::%RUN-PROGRAM abstracts the Lisp implementations capable of asynchronous operation, I guess it is a little more stable.
The interface to UIOP::%RUN-PROGRAM is rather stable at this point, though I've refactored its internals recently. For historical reasons, it isn't exported — it used to be quite uglier and more limited than it is now.
I believe it's OK to use it, although we may or may not rename it if and when we officially export it (and may or may not provide a smooth transition, depending on whether anything in quicklisp depends on it).
If you only care about CCL and SBCL, you can use UIOP::%RUN-PROGRAM.
If you want a portable solution that requires an external library, try IOLIB/OS:CREATE-PROCESS.
The "external library" library part of IOLIB always tends to make me shy away.
That's a valid reason to stick with UIOP::%RUN-PROGRAM.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If prices are information, then subsidies are censorship. — Russ Nelson
-
Faré -
Mark Evenson -
Stelian Ionescu