The first snapshot of the year has been tagged, finally[1]. Binaries will be uploaded soon. Perhaps the most important news is that the repro for cmucl has been moved to https://gitlab.common-lisp.net/cmucl/cmucl, along with most of common-lisp.net. The old git repro is gone and has been replaced with this new one. The current trac instance is still running and will remain so, but we are planning on migrating everything from trac to gitlab. Please use gitlab to file new issues. The other changes to cmucl include: o The UNIX package has been changed; it now only contains just enough to compile all of cmucl. If you want the rest of old UNIX package, use (require :unix) to get that. o The clx-inspector contrib module has been added, courtesy of Fred Gilham. o ASDF documentation in html, info, and pdf formats is included. o Issues fixed: o gitlab issue #1 fixed. -- Ray Footnotes: [1] I've been occupied with some other things of late, some related to cmucl and some not.
The other changes to cmucl include:
o The UNIX package has been changed; it now only contains just enough to compile all of cmucl. If you want the rest of old UNIX package, use (require :unix) to get that.
1- This breaks ASDF, which assumed it could use unix-getenv and some such. 2- (require :unix) doesn't work in the snapshot. a- the module is named unix, but the fasl file is called unix-glibc2.sse2f, whether that's the reason or not, I get: 0] (require :unix) File-error in function LISP::INTERNAL-LOAD: "modules:load-unix" does not exist. Error flushed ... b- if I try to load the file manually, I get a package lock error: 0] (load "/home/tunes/local/stow/cmucl/lib/cmucl/lib/contrib/unix/unix-glibc2.sse2f") ; Loading #P"/home/tunes/local/stow/cmucl/lib/cmucl/lib/contrib/unix/unix-glibc2.sse2f". Attempt to modify the locked package UNIX, by redefining function (SETF GROUP-INFO-NAME) Error flushed ...
o Issues fixed: o gitlab issue #1 fixed. Should I be using gitlab to report issues from now on?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He who says he will die for a cause will probably lie for it and may kill for it. — John McCarthy
"Fare" == Far <Far> writes:
>> The other changes to cmucl include: >> >> o The UNIX package has been changed; it now only contains just >> enough >> to compile all of cmucl. If you want the rest of old UNIX >> package, use (require :unix) to get that. >> Fare> 1- This breaks ASDF, which assumed it could use unix-getenv Fare> and some such. 2- (require :unix) doesn't work in the Fare> snapshot. Fare> a- the module is named unix, but the fasl file is called Fare> unix-glibc2.sse2f, Fare> whether that's the reason or not, I get: Fare> 0] (require :unix) Fare> File-error in function LISP::INTERNAL-LOAD: Fare> "modules:load-unix" does not exist. Error flushed ... Fare> b- if I try to load the file manually, I get a package Fare> lock error: Fare> 0] (load Fare> "/home/tunes/local/stow/cmucl/lib/cmucl/lib/contrib/unix/unix-glibc2.sse2f") Fare> ; Loading Fare> ; #P"/home/tunes/local/stow/cmucl/lib/cmucl/lib/contrib/unix/unix-glibc2.sse2f". Fare> Attempt to modify the locked package UNIX, by Fare> redefining function (SETF GROUP-INFO-NAME) Fare> Error flushed ... Dang. I must have missed adding load-unix to the distribution. load-unix takes care of unlocking the UNIX package, but perhaps unix-glibc2 should do that itself. The fact that the file is unix-glibc2 is an implementation detail where unix.lisp is used everywhere except on linux for historical reasons. I'll add unix-getenv to the core in the next snapshot. I'll take a look through asdf to see if there are other items from the UNIX package that are needed. Sorry for the trouble and thanks for the report. >> o Issues fixed: >> o gitlab issue #1 fixed. Fare> Should I be using gitlab to report issues from now on? That's the preferred way so it doesn't get lost, but the mailing list is ok too. -- Ray
"Fare" == Far <Far> writes:
>> The other changes to cmucl include: >> >> o The UNIX package has been changed; it now only contains just >> enough >> to compile all of cmucl. If you want the rest of old UNIX >> package, use (require :unix) to get that. >> Fare> 1- This breaks ASDF, which assumed it could use unix-getenv Fare> and some such. 2- (require :unix) doesn't work in the Fare> snapshot. Fare> a- the module is named unix, but the fasl file is called Fare> unix-glibc2.sse2f, Fare> whether that's the reason or not, I get: Fare> 0] (require :unix) Fare> File-error in function LISP::INTERNAL-LOAD: Fare> "modules:load-unix" does not exist. Error flushed ... Can you tell me how you did this? I just grabbed the tarballs and tried it out on a linux box. (require :unix) works for me. -- Ray
"Fare" == Far <Far> writes:
>> The other changes to cmucl include: >> >> o The UNIX package has been changed; it now only contains just enough >> to compile all of cmucl. If you want the rest of old UNIX package, >> use (require :unix) to get that. >> Fare> 1- This breaks ASDF, which assumed it could use unix-getenv and some such. I looked at the asdf distributed with cmucl, which is version 3.1.4. asdf does have getenv, but not via unix-getenv. Instead it accesses ext:*environment-list*. A quick grep through asdf.lisp only shows unix:unix-current-directory, unix:unix-chdir, unix:unix-stat, and unix:unix-rmdir, which are all available without doing (require :unix). Is there some way for me to test asdf? (I confess I haven't run the asdf test suite with this new snapshot. I should have.) However, I do see that I've broken slime which wants to use unix:unix-execve, and perhaps others. This is fixed by adding (require :unix) in my .cmucl-init.lisp script, but that's not really the solution I want. -- Ray
On Wed, Jun 10, 2015 at 1:55 PM, Raymond Toy <toy.raymond@gmail.com> wrote:
"Fare" == Far <Far> writes:
>> The other changes to cmucl include: >> >> o The UNIX package has been changed; it now only contains just enough >> to compile all of cmucl. If you want the rest of old UNIX package, >> use (require :unix) to get that. >> Fare> 1- This breaks ASDF, which assumed it could use unix-getenv and some such.
I looked at the asdf distributed with cmucl, which is version 3.1.4. asdf does have getenv, but not via unix-getenv. Instead it accesses ext:*environment-list*.
The latest 3.1.4.15 uses unix-getenv. I can revert that, if you wish.
A quick grep through asdf.lisp only shows unix:unix-current-directory, unix:unix-chdir, unix:unix-stat, and unix:unix-rmdir, which are all available without doing (require :unix).
Is there some way for me to test asdf? (I confess I haven't run the asdf test suite with this new snapshot. I should have.)
cd asdf ; make t l=cmucl
However, I do see that I've broken slime which wants to use unix:unix-execve, and perhaps others. This is fixed by adding (require :unix) in my .cmucl-init.lisp script, but that's not really the solution I want.
BTW, I retried running cmucl after I rm rf lib/cmucl and tar jxf the 2015-06 tarball, and I can 100% reproduce the failure to (require :unix). Are you extracting on top of an installation that has extra files? CMU Common Lisp snapshot-2015-06 (20F Unicode), running on lux With core: /home/tunes/local/stow/cmucl/lib/cmucl/lib/lisp-sse2.core Dumped on: Thu, 2015-06-04 23:17:46-04:00 on lorien3 See <http://www.cmucl.org/> for support information. Loaded subsystems: Unicode 1.29 with Unicode version 6.2.0 Python 1.1, target Intel x86/sse2 CLOS based on Gerd's PCL 2010/03/19 15:19:03 * (require :unix) File-error in function LISP::INTERNAL-LOAD: "modules:load-unix" does not exist. [Condition of type KERNEL:SIMPLE-FILE-ERROR] Restarts: 0: [CHECK-AGAIN] See if it exists now. 1: [USE-VALUE ] Prompt for a new name. 2: [CONTINUE ] Return NIL from load of "modules:load-unix". 3: [ABORT ] Return to Top-Level. Debug (type H for help) (LISP::INTERNAL-LOAD #P"modules:load-unix" NIL :ERROR NIL ...) Source: Error finding source: Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists: target:code/load.lisp. 0] #p"modules:" #P"modules:" 0] (describe #p"modules:") #P"modules:" is a structure of type PATHNAME. HOST: #<LISP::UNIX-HOST>. DEVICE: NIL. DIRECTORY: (:ABSOLUTE #<SEARCH-LIST modules>). NAME: NIL. TYPE: NIL. VERSION: NIL. What kind of pathname horror is THAT????? As if pathnames weren't horrible enough already, CMUCL invents additional pathname horrors! —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Fraud is the homage that force pays to reason. — Charles Curtis
"Fare" == Far <Far> writes:
Fare> On Wed, Jun 10, 2015 at 1:55 PM, Raymond Toy <toy.raymond@gmail.com> wrote: >>>>>>> "Fare" == Far <Far> writes: >> >> >> The other changes to cmucl include: >> >> >> >> o The UNIX package has been changed; it now only contains just enough >> >> to compile all of cmucl. If you want the rest of old UNIX package, >> >> use (require :unix) to get that. >> >> Fare> 1- This breaks ASDF, which assumed it could use unix-getenv and some such. >> >> I looked at the asdf distributed with cmucl, which is version >> 3.1.4. asdf does have getenv, but not via unix-getenv. Instead it >> accesses ext:*environment-list*. >> Fare> The latest 3.1.4.15 uses unix-getenv. I can revert that, if you wish. What would you prefer? I'm open to adding back unix-getenv. Since cmucl tracks asdf fairly closely, it seems either would be ok. >> A quick grep through asdf.lisp only shows unix:unix-current-directory, >> unix:unix-chdir, unix:unix-stat, and unix:unix-rmdir, which are all >> available without doing (require :unix). >> >> Is there some way for me to test asdf? (I confess I haven't run the >> asdf test suite with this new snapshot. I should have.) >> Fare> cd asdf ; make t l=cmucl >> However, I do see that I've broken slime which wants to use >> unix:unix-execve, and perhaps others. This is fixed by adding >> (require :unix) in my .cmucl-init.lisp script, but that's not really >> the solution I want. >> Fare> BTW, I retried running cmucl after I rm rf lib/cmucl and tar jxf the Fare> 2015-06 tarball, and I can 100% reproduce the failure to (require Ah, you also need the 2015-06 extra tarball since that contains the contrib stuff. Since this is a compatibility issue, it's probably best if the normal tarball contained these files. I'll make that happen for the next snapshot. Fare> (LISP::INTERNAL-LOAD #P"modules:load-unix" NIL :ERROR NIL ...) Fare> Source: Error finding source: Fare> Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists: Fare> target:code/load.lisp. Fare> 0] #p"modules:" Fare> #P"modules:" Fare> 0] (describe #p"modules:") Fare> #P"modules:" is a structure of type PATHNAME. Fare> HOST: #<LISP::UNIX-HOST>. Fare> DEVICE: NIL. Fare> DIRECTORY: (:ABSOLUTE #<SEARCH-LIST modules>). Fare> NAME: NIL. Fare> TYPE: NIL. Fare> VERSION: NIL. Fare> What kind of pathname horror is THAT????? Fare> As if pathnames weren't horrible enough already, CMUCL invents Fare> additional pathname horrors! You clearly have not been using cmucl for very long. :-) That search-list pathname stuff predates my involvement with cmucl. Could possibly predate support for logical pathnames. It certainly would be better if the host were something like #<lisp::search-list-host "modules"> and the :directory component could be more like a regular list of directory components. I did try to make that happen briefly once but failed to succeed. And decided that it wasn't worth it to me. -- Ray
On Wed, Jun 10, 2015 at 5:51 PM, Raymond Toy <toy.raymond@gmail.com> wrote:
"Fare" == Far <Far> writes:
Fare> On Wed, Jun 10, 2015 at 1:55 PM, Raymond Toy <toy.raymond@gmail.com> wrote: >>>>>>> "Fare" == Far <Far> writes: >> >> >> The other changes to cmucl include: >> >> >> >> o The UNIX package has been changed; it now only contains just enough >> >> to compile all of cmucl. If you want the rest of old UNIX package, >> >> use (require :unix) to get that. >> >> Fare> 1- This breaks ASDF, which assumed it could use unix-getenv and some such. >> >> I looked at the asdf distributed with cmucl, which is version >> 3.1.4. asdf does have getenv, but not via unix-getenv. Instead it >> accesses ext:*environment-list*. >> Fare> The latest 3.1.4.15 uses unix-getenv. I can revert that, if you wish.
What would you prefer? I'm open to adding back unix-getenv. Since cmucl tracks asdf fairly closely, it seems either would be ok.
ext:*environment-list* is O(n); it sucks and is totally clunky, and not case-preserving. Kill it with fire.
>> A quick grep through asdf.lisp only shows unix:unix-current-directory, >> unix:unix-chdir, unix:unix-stat, and unix:unix-rmdir, which are all >> available without doing (require :unix). >> >> Is there some way for me to test asdf? (I confess I haven't run the >> asdf test suite with this new snapshot. I should have.) >> Fare> cd asdf ; make t l=cmucl
>> However, I do see that I've broken slime which wants to use >> unix:unix-execve, and perhaps others. This is fixed by adding >> (require :unix) in my .cmucl-init.lisp script, but that's not really >> the solution I want. >>
Fare> BTW, I retried running cmucl after I rm rf lib/cmucl and tar jxf the Fare> 2015-06 tarball, and I can 100% reproduce the failure to (require
Ah, you also need the 2015-06 extra tarball since that contains the contrib stuff. Since this is a compatibility issue, it's probably best if the normal tarball contained these files. I'll make that happen for the next snapshot.
It doesn't make sense for the normal tarball to contain asdf but not some module required for asdf. Maybe add a canary test for your tarballs, that makes sure you can run some trivial asdf program?
Fare> (LISP::INTERNAL-LOAD #P"modules:load-unix" NIL :ERROR NIL ...) Fare> Source: Error finding source: Fare> Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists: Fare> target:code/load.lisp. Fare> 0] #p"modules:"
Fare> #P"modules:" Fare> 0] (describe #p"modules:")
Fare> #P"modules:" is a structure of type PATHNAME. Fare> HOST: #<LISP::UNIX-HOST>. Fare> DEVICE: NIL. Fare> DIRECTORY: (:ABSOLUTE #<SEARCH-LIST modules>). Fare> NAME: NIL. Fare> TYPE: NIL. Fare> VERSION: NIL.
Fare> What kind of pathname horror is THAT????? Fare> As if pathnames weren't horrible enough already, CMUCL invents Fare> additional pathname horrors!
You clearly have not been using cmucl for very long. :-) That search-list pathname stuff predates my involvement with cmucl. Could possibly predate support for logical pathnames.
It certainly would be better if the host were something like #<lisp::search-list-host "modules"> and the :directory component could be more like a regular list of directory components. I did try to make that happen briefly once but failed to succeed. And decided that it wasn't worth it to me.
Yup, probably not worth stirring the ancient cesspool and risking to awaken a great old one. Thanks for your support! —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Reason wins in the long run, because irrational memes fight each other, whereas rational memes add up.
On Thu, Jun 11, 2015 at 1:31 AM, Carl Shapiro <carl.shapiro@gmail.com> wrote:
On Wed, Jun 10, 2015 at 3:02 PM, Faré <fahree@gmail.com> wrote:
ext:*environment-list* is O(n); it sucks and is totally clunky, and not case-preserving. Kill it with fire.
What would you propose replacing it with?
Lazy getenv, putenv — and if you want to get all of them, or to pass all of them, use an equal hash-table with strings as key, not an alist with keywords as keys. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org So capitalism has been making everyone poorer for centuries? How fabulously rich our ancestors must have been before all this destruction!
On Wed, Jun 10, 2015 at 11:09 PM, Faré <fahree@gmail.com> wrote:
Lazy getenv, putenv — and if you want to get all of them, or to pass all of them, use an equal hash-table with strings as key, not an alist with keywords as keys.
Thanks for the clarification. Is your issue with the search time? The libc getenv and putenv uses a linear search with a linear time compare of strings. It is unlikely to be faster than an a-list search. I agree that converting strings to keywords is problematic when dealing with unconventionally cased environment variable names. The original strings are kept around if you need them.
On Thu, Jun 11, 2015 at 2:40 AM, Carl Shapiro <carl.shapiro@gmail.com> wrote:
On Wed, Jun 10, 2015 at 11:09 PM, Faré <fahree@gmail.com> wrote:
Lazy getenv, putenv — and if you want to get all of them, or to pass all of them, use an equal hash-table with strings as key, not an alist with keywords as keys.
Thanks for the clarification.
Is your issue with the search time? The libc getenv and putenv uses a linear search with a linear time compare of strings. It is unlikely to be faster than an a-list search.
My issue is that ext:*environment-list* is an abstraction inversion. Instead of providing a higher-level view on an existing API, it reinvents a new low-level data structure that doesn't provide any performance advantage (indeed, only slowdown) yet introduces a new semantic gap (albeit small) between what was and what is, and only gets in the way of a further API improvement. But if you declare it the official way to access the environment in cmucl, I'll revert to using it.
I agree that converting strings to keywords is problematic when dealing with unconventionally cased environment variable names. The original strings are kept around if you need them.
At this point, it's bikeshedding. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Microphones, liberating singers from having to be loud to be heard, gave back to each language a music that matches its own prosody.
"Fare" == Far <Far> writes:
Fare> On Thu, Jun 11, 2015 at 2:40 AM, Carl Shapiro <carl.shapiro@gmail.com> wrote: >> On Wed, Jun 10, 2015 at 11:09 PM, Faré <fahree@gmail.com> wrote: >>> >>> Lazy getenv, putenv — and if you want to get all of them, or to pass >>> all of them, use an equal hash-table with strings as key, not an alist >>> with keywords as keys. >> >> Thanks for the clarification. >> >> Is your issue with the search time? The libc getenv and putenv uses a >> linear search with a linear time compare of strings. It is unlikely to be >> faster than an a-list search. >> Fare> My issue is that ext:*environment-list* is an abstraction inversion. Fare> Instead of providing a higher-level view on an existing API, it Fare> reinvents a new low-level data structure that doesn't provide any Fare> performance advantage (indeed, only slowdown) yet introduces a new Fare> semantic gap (albeit small) between what was and what is, and only Fare> gets in the way of a further API improvement. My assumption is that it exists so that execve or whatever can be called with a new environment that can be easily manipulated from lisp. Fare> But if you declare it the official way to access the environment in Fare> cmucl, I'll revert to using it. I will put back unix-getenv into the core. And I'll update the main tarball to include the unix contrib. -- Ray
On Thu, Jun 11, 2015 at 8:53 AM, Raymond Toy <toy.raymond@gmail.com> wrote:
My assumption is that it exists so that execve or whatever can be called with a new environment that can be easily manipulated from lisp.
Correct. The CMUCL behavior of copying the environment into a native associative structure at start-up (and using that copy to seed the environment of new subprocesses) is very common. Here are some examples https://docs.python.org/2/library/os.html http://golang.org/pkg/os/#Setenv http://docs.oracle.com/javase/7/docs/api/java/lang/System.html There is no consensus regarding what happens to the libc environment when the environment copy is modified. Library designers seem to prefer speed of environment access to consistency with the libc environment. It is not obvious whether there are significant benefits to doing anything differently, but if there is a case to be made for a different behavior, it would be helpful to know. That would provide a basis for a technical discussion.
On Thu, Jun 11, 2015 at 8:53 AM, Raymond Toy <toy.raymond@gmail.com> wrote:
My assumption is that it exists so that execve or whatever can be called with a new environment that can be easily manipulated from lisp.
Correct.
The CMUCL behavior of copying the environment into a native associative structure at start-up (and using that copy to seed the environment of new subprocesses) is very common.
You can have both, which IMO is the right thing wrt. to backwards-compatibility with C programs. With IOlib one can query the process-level environment or turn that into an object, manipulate it and pass it to CREATE-PROCESS. See https://github.com/sionescu/iolib/blob/master/src/os/os-unix.lisp#L10. Additionally, since a C environment is an array of pointers, it's probably still faster to query if speed is a concern. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib
On Fri, Jun 12, 2015 at 6:52 AM, Stelian Ionescu <sionescu@cddr.org> wrote:
You can have both, which IMO is the right thing wrt. to backwards-compatibility with C programs.
Interesting. But, how exactly would this work? The C library provides no signal to invalidate a cache when its copy of the environment changes As such, it seems impossible to efficiently keep a copy of the environment in Lisp consistent. Without consistency, I am not sure how backward compatibility is maintained.
With IOlib one can query the process-level environment or turn that into an object, manipulate it and pass it to CREATE-PROCESS. See https://github.com/sionescu/iolib/blob/master/src/os/os-unix.lisp#L10.
Thanks for the pointer. I noticed that your hash-table construction does not check for the preexistence of a value when doing inserts from the environment list (on line 101). While uncommon, this nevertheless is a discrepancy between the behavior of getenv in your library and the and getenv in the C library. This can create a security issue.
Additionally, since a C environment is an array of pointers, it's probably still faster to query if speed is a concern.
An array of pointers will certainly remove the overhead of the linked-list traversal. However, the EQUAL compare of the referent might eliminate the savings. This would have to be measured.
On Fri, Jun 12, 2015 at 6:52 AM, Stelian Ionescu <sionescu@cddr.org> wrote:
You can have both, which IMO is the right thing wrt. to backwards-compatibility with C programs.
Interesting. But, how exactly would this work? The C library provides no signal to invalidate a cache when its copy of the environment changes As such, it seems impossible to efficiently keep a copy of the environment in Lisp consistent. Without consistency, I am not sure how backward compatibility is maintained.
There is no "its copy", just a single array that putenv() changes destructively. During an exec(), the runtime linker allocates that array, having a fixed maximum size baked in the kernel, at a fixed offset in the process address space. If something modifies that in-place and screws it up, there's no backup.
With IOlib one can query the process-level environment or turn that into an object, manipulate it and pass it to CREATE-PROCESS. See https://github.com/sionescu/iolib/blob/master/src/os/os-unix.lisp#L10.
Thanks for the pointer. I noticed that your hash-table construction does not check for the preexistence of a value when doing inserts from the environment list (on line 101). While uncommon, this nevertheless is a discrepancy between the behavior of getenv in your library and the and getenv in the C library. This can create a security issue.
Thanks for the notice. I opened https://github.com/sionescu/iolib/issues/24 -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib
"Far" == Far <Far> writes:
Far> On Wed, Jun 10, 2015 at 5:51 PM, Raymond Toy <toy.raymond@gmail.com> wrote: >>>>>>> "Fare" == Far <Far> writes: >> Fare> On Wed, Jun 10, 2015 at 1:55 PM, Raymond Toy <toy.raymond@gmail.com> wrote: >> >>>>>>> "Fare" == Far <Far> writes: >> >> >> >> >> The other changes to cmucl include: >> >> >> >> >> >> o The UNIX package has been changed; it now only contains just enough >> >> >> to compile all of cmucl. If you want the rest of old UNIX package, >> >> >> use (require :unix) to get that. >> >> >> Fare> 1- This breaks ASDF, which assumed it could use unix-getenv and some such. >> >> >> >> I looked at the asdf distributed with cmucl, which is version >> >> 3.1.4. asdf does have getenv, but not via unix-getenv. Instead it >> >> accesses ext:*environment-list*. >> >> Fare> The latest 3.1.4.15 uses unix-getenv. I can revert that, if you wish. >> >> What would you prefer? I'm open to adding back unix-getenv. Since >> cmucl tracks asdf fairly closely, it seems either would be ok. >> Far> ext:*environment-list* is O(n); it sucks and is totally clunky, and Far> not case-preserving. Kill it with fire. I submit that if the O(n) behavior is observable in asdf, you're doing it wrong. :-) And it is case preserving. Thus, the envvar "foo" would be the keyword :|foo|, and so on. -- Ray
participants (4)
-
Carl Shapiro -
Faré -
Raymond Toy -
Stelian Ionescu