I like the portable getenv implementation - Not having that available is why my proposal uses a logical pathname so that one can define the temporary directory easily from the outside.

Both asdf and xcvb-driver export getenv.

Then again, I'm not sure that Alexandria should include loads of implementation specific switches.  It might then be better to combine a cross-implementation getenv and the temporary file handling stuff into a trivial-temporary-file library and let Quicklisp do the rest.

If you're unwilling to handle non-trivial system dependencies in alexandria, then maybe alexandria is not the place for this particular piece of code. Osicat, iolib (why do we need two such libraries?) might be better suited, or you could use xcvb-driver that exports this piece of functionality.

OT: Zach, I made many related changes in asdf, fare-utils, xcvb, recently, and pushed working combinations yesterday. I also still haven't implemented self-upgrade of xcvb, which is required in case the cached xcvb executable is older than your xcvb driver needs. But at least, it will error out loudly now.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Not all Law is created equal before Man. Some Law causes least conflict and least perverse incentives. By definition we call it Natural Law.

2012/4/22 Hans Hübner hans.huebner@gmail.com:

2012/4/22 Faré fahree@gmail.com:

...

...

I like the portable getenv implementation - Not having that available is why my proposal uses a logical pathname so that one can define the temporary directory easily from the outside.

Both asdf and xcvb-driver export getenv.

I am aware of that.  I am lobbying for the inclusion in Alexandria so that libraries which need the functionality need not depend on a build system for this basic, non build-related functionality.  The idea was motivated by work someone does on Postmodern, and even though Quicklisp exists, having Postmodern depend on either XCVB or ASDF does not seem like The Right Thing.

Unless postmodern can be loaded without ASDF, I don't see a problem with making its dependency on ASDF explicit. Also, xcvb-driver itself is not a build system (xcvb is), it's a basic runtime, providing portable abstractions from the underlying system. XCVB has a strict dependency on xcvb-driver, but the reverse dependency is looser. Maybe that would be more obvious if I split its repository from that of xcvb itself?

...

If you're unwilling to handle non-trivial system dependencies in alexandria, then maybe alexandria is not the place for this particular piece of code. Osicat, iolib (why do we need two such libraries?) might be better suited, or you could use xcvb-driver that exports this piece of functionality.

Neither osicat nor iolib offer portable implementations for temporary file name generation.

My point is that you should offer a patch to these systems, not to alexandria, if you are to depend on functionality only these systems can provide to do the Right Thing(tm) with respect to temporary files. And if only for security purposes, that seems like a good idea: if you introducing a potential security issue in a basic library, that gets inherited implicitly by all innocent users of a poorly written piece of code, that's a poor way to promote Lisp as a serious language in which to write serious software. I'd rather any widely promoted version of with-temporary-file either use mkstemp with appropriate defaults for the directory, or try very hard to behave in a similar fashion to mkostemp with appropriate defaults. Unhappily, setting the umask around an otherwise umask-ignorant with-temporary-file is not thread-safe.

For well-delimited purposes such as in xcvb-driver, trusting the umask might be OK, but for a basic foundational library such as alexandria, I believe it's inviting disaster. Please submit an enhancement to iolib or osicat instead of alexandria.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org License Agreement: By reading this message, you agree to run around the room which you are currently in, flapping your arms, and squawking like a chicken.

On Sun, Apr 22, 2012 at 1:55 PM, Antony lisp.linux@gmail.com wrote:

In addition to the suggested use of mkstemp kind of things, I'd like to point out that your proposed code's use of 'random' may not be thread safe (I don't mean it's hazardous, but it may not generate unique names).  May not be a big deal, but just one more point to consider.

Thank you for pointing that out. I'm not quite sure whether one can expect calls to be thread-safe or not. The standard certainly has nothing to say about this, but maybe someone here knows what the implementations do?

Thanks, Hans

Hans Hübner hans.huebner@gmail.com writes:

Faré,

neither iolib nor osicat are the right place for temporary file handling operators like the one I proposed. Both these libraries are specifically targeted towards POSIX systems, so there is no chance that they will cooperate, say, with ABCL.

A lot of systems implement the POSIX API. http://en.wikipedia.org/wiki/POSIX

Does the JVM run on a non-POSIX plateform? Does it not provide the mkstemp(3) POSIX function?

Setting the process umask is something that is completely unrelated to portably creating temporary files.

You mean to say that (CL) portably creating temporary files cannot be done controling the umask, because there's no such notion in CL.

That's all right, and for a lot of uses, this may be OK, but that doesn't fulfill the need for a mkstemp() when the security consideration need to be taken into account, ie. as soon as you're writing a real program.

Ie. there's no point in providing a conforming temporary file creation, for real stuff you need a portability temporary file creation library that allows the setting of umask.

If this is not going into Alexandria, there are two likely other outcomes: Either the code goes into Postmodern, giving the next person the same problem (and I've not had this problem once, but many times), or it will end up in a trivial-temporary-file library that may include some conditional code for setting up the default temporary directory in a meaningful way. With Quicklisp, this would not hurt much, but it certainly feels a little silly to additional trivial-whatever libraries for basic functionality.

You're discovering that creation of temporary files is not so trivial after all. On unix, this has a long history of tries and failures. To wit:

mktemp(3)

BUGS

Never use mktemp(). Some implementations follow 4.3BSD and replace XXXXXX by the current process ID and a single letter, so that at most 26 different names can be returned. Since on the one hand the names are easy to guess, and on the other hand there is a race between testing whether the name exists and opening the file, every use of mktemp() is a security risk. The race is avoided by mkstemp(3).

tempnam(3)

NOTES

Although tempnam() generates names that are difficult to guess, it is nevertheless possible that between the time that tempnam() returns a pathname, and the time that the program opens it, another program might create that pathname using open(2), or create it as a symbolic link. This can lead to security holes. To avoid such possi‐ bilities, use the open(2) O_EXCL flag to open the pathname. Or better yet, use mkstemp(3) or tmpfile(3).

SUSv2 does not mention the use of TMPDIR; glibc will use it only when the program is not set-user-ID. On SVr4, the directory used under d) is /tmp (and this is what glibc does).

Because it dynamically allocates memory used to return the pathname, tempnam() is reentrant, and thus thread safe, unlike tmpnam(3).

The tempnam() function generates a different string each time it is called, up to TMP_MAX (defined in <stdio.h>) times. If it is called more than TMP_MAX times, the behavior is implementation defined.

tempnam() uses at most the first five bytes from pfx.

The glibc implementation of tempnam() will fail with the error EEXIST upon failure to find a unique name.

BUGS

The precise meaning of "appropriate" is undefined; it is unspecified how accessibil‐ ity of a directory is determined.

Never use this function. Use mkstemp(3) or tmpfile(3) instead.

mkstemp(3)

NOTES

The old behavior of creating a file with mode 0666 may be a security risk, especially since other UNIX flavors use 0600, and somebody might overlook this detail when porting pro‐ grams.

More generally, the POSIX specification of mkstemp() does not say anything about file modes, so the application should make sure its file mode creation mask (see umask(2)) is set appropriately before calling mkstemp() (and mkostemp()).

So it's not the first time that such difficulties are identified in an apparently simple API.

Again, for a toy, just creating a randomly named file is ok. But in production code you will need more than that.

Your remarks regarding how this request represents a "poor way to promote Lisp as a serious language" are unwelcome, unconstructive and wrong. Thank you for not continuing on that path.

That's a little strong.

I'm coming on the side of saying that this has a place in Madeira, (first version of which I'm releasing before the end of May.)

Madeira can deal with the umask and thread-safety of *random-state*, whereas Alexandria has so far resisted unportable or implementation specific code -- and dealing with those two requires that.

Umask could be degreed to simply be the process umask, which would be suboptimal but acceptable, but thread-safety is more serious -- though I suspect that parallel RANDOM calls should at worst produce degraded quality of random numbers, not corruption or errors, but every time someone assumes "this can't cause serious issues" a kitten dies, so...

Cheers,

-- nikodemus

On Wed, 2012-04-25 at 06:26 -0400, Hans Hübner wrote:

Dealing with the umask has no place in this library. "umask" is a system-specific concept, and file security is something that is orthogonal to temporary files. If an application wishes to give its temporary files different permissions than other files that it creates, it should do so explicitly.

Quite the opposite, the permissions must be specified at creation time, otherwise it's useless

And besides, why put temporary files into a shared directory in the first place?

Old habits(a.k.a. Unix tradition). Distributions are slowly changing this, but a Lisp library should work in any case

The right way to deal with possible issues in this respect is to create an application specific temporary directory that is completely protected before starting the application, and communicating that directory to the application by the way of the TEMP or TMPDIR environment variable. That is how Windows and OSX do it, and it is much better than trying to protect files in a shared /tmp/ directory. As if we did not have a hierarchical file system for a few decades.

That's not how they do it. Windows and OSX have a user-specific temporary directory and applications usually create a subdirectory of their own but with a well-known name: see documentation for NSTemporaryDirectory, for example

On Wed, 2012-04-25 at 09:05 -0400, Hans Hübner wrote:

On Wed, Apr 25, 2012 at 8:42 AM, Stelian Ionescu sionescu@cddr.org wrote:

...

On Wed, 2012-04-25 at 06:26 -0400, Hans Hübner wrote:

...

Dealing with the umask has no place in this library. "umask" is a system-specific concept, and file security is something that is orthogonal to temporary files. If an application wishes to give its temporary files different permissions than other files that it creates, it should do so explicitly.

Quite the opposite, the permissions must be specified at creation time, otherwise it's useless

If the Lisp implementation provides a means to specify the file permissions to cl:open, then that would be how it should be done for temporary files, too. I do not understand what you mean by "quite the opposite" here. Is there anything in the proposed API that makes it impossible to specify file permissions, if so supported by the underlying implementation?

To achieve safety, the umask must be specified on creation. If cl:open doesn't allow that, then it's effectively useless for the purpose of properly creating temporary files

...

That's not how they do it. Windows and OSX have a user-specific temporary directory and applications usually create a subdirectory of their own but with a well-known name: see documentation for NSTemporaryDirectory, for example

Windows and OSX do better than Unix by default in that they have user specific temporary directories, and this is all that we're discussing anyway. The only isolation that one can hope to have is on a per-user level, i.e. one may want to prevent that a process running under a some user ID can access a (temporary or non-temporary) file created by a process running under a different user ID. By making the user-specific temporary directories inaccessible to other user IDs, this goal is neatly achieved.

The goal should be to protect, as much as possible, also against malicious programs running under the same UID and randomizing file and directory names, as opposed to using well-known names, is very useful in that regard

2012/4/22 Hans Hübner hans.huebner@gmail.com:

neither iolib nor osicat are the right place for temporary file handling operators like the one I proposed.  Both these libraries are specifically targeted towards POSIX systems, so there is no chance that they will cooperate, say, with ABCL.

iolib certainly *intends* to be more portable than that, but admittedly isn't quite there yet.

Setting the process umask is something that is completely unrelated to portably creating temporary files.

The umask, atomicity and a proper directory are all important security concerns. An implementation that doesn't make sure these issues are handled should make sure to tag itself as "not for use where security matters" AND to encourage all its clients to taint themselves as such, too.

Failure of my implementation to properly address security concerns (for the same reason as yours) is one reason I haven't pushed my own with-temporary-file for semi-standardization in the past. I still believe its API is better than the one you propose (though I can imagine further improvements for creation modes, early deletion, etc.).

If this is not going into Alexandria, there are two likely other outcomes:  Either the code goes into Postmodern, giving the next person the same problem (and I've not had this problem once, but many times), or it will end up in a trivial-temporary-file library that may include some conditional code for setting up the default temporary directory in a meaningful way. With Quicklisp, this would not hurt much, but it certainly feels a little silly to additional trivial-whatever libraries for basic functionality.

Please go the trivial-temporary-file way. It is not silly at all.

Your remarks regarding how this request represents a "poor way to promote Lisp as a serious language" are unwelcome, unconstructive and wrong.  Thank you for not continuing on that path.

Reinventing an octogonal wheel then trying to semi-standardize it is poor taste. Please either use existing round wheels, or make it round if you make a new one. Please no 90% solutions that come bite you later; or at the very least, document it as a 90% solution and discourage its use in code that aims at providing 100% abstractions.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Truth comes as conqueror only to those who have lost the art of receiving it as friend. — Tagore