[Cffi-devel] Pre-populate cffi:*foreign-library-directories* by OS
From time to time I have seen requests from users to include an absolute path (starting from "/") in systems that use load-foreign-library under a clause for their favorite OS. This makes me nervous, especially when it is a little-used OS, because I have the feeling the requester's configuration is not typical for that OS and I will later get a request to add a different path when the original requester has moved on.
Ideally, there would never have to be any absolute paths; dlopen would always know where to find libraries, because the OS is always configured in a standard way. However, a quick survey of this topic shows that reality falls far short of this ideal, and remedies are not clear. It is well beyond our capabilities to fix the entire world on this matter. Between fixing the world and fixing every CFFI-using application one by one, there is the compromise of setting default search paths for each OS in CFFI itself, thereby opening all applications to proper functionality out of the box for most OSes. The variable cffi:*foreign-library-directories* seems like the right thing to set. I've looked through all Quicklisp libraries for absolute paths in uses of load-foreign-library, and found these: Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib Therefore I propose to change the definition to: (defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.") As requests come in to add an absolute path for an application, they can be referred to this mailing list to request the change here, if it is not an application-specific path. Then it is more likely to be properly vetted for general applicability for all users of that OS, and will be available for all libraries. Does this sound like a reasonable way to handle this problem? Liam
Hi Liam, as it does not hurt to have multiple directories in *foreign-library-directories* I personally don't care what the entries are ... One of the first things I do is set it to nil .. The OS does know how to load libs. If someone wants to add dirs then please do this via env vars!! Just my two cents. Cheers Frank PS I don't understand all the discussion around this. Proper env var setting just about cures everything. Von meinem iPhone gesendet
Am 30.03.2014 um 15:41 schrieb Liam Healy <lnp@healy.washington.dc.us>:
From time to time I have seen requests from users to include an absolute path (starting from "/") in systems that use load-foreign-library under a clause for their favorite OS. This makes me nervous, especially when it is a little-used OS, because I have the feeling the requester's configuration is not typical for that OS and I will later get a request to add a different path when the original requester has moved on.
Ideally, there would never have to be any absolute paths; dlopen would always know where to find libraries, because the OS is always configured in a standard way. However, a quick survey of this topic shows that reality falls far short of this ideal, and remedies are not clear. It is well beyond our capabilities to fix the entire world on this matter.
Between fixing the world and fixing every CFFI-using application one by one, there is the compromise of setting default search paths for each OS in CFFI itself, thereby opening all applications to proper functionality out of the box for most OSes. The variable cffi:*foreign-library-directories* seems like the right thing to set. I've looked through all Quicklisp libraries for absolute paths in uses of load-foreign-library, and found these:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
As requests come in to add an absolute path for an application, they can be referred to this mailing list to request the change here, if it is not an application-specific path. Then it is more likely to be properly vetted for general applicability for all users of that OS, and will be available for all libraries. Does this sound like a reasonable way to handle this problem?
Liam _______________________________________________ Cffi-devel mailing list Cffi-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
Thanks Frank. Unfortunately it is just not true that "The OS does know how to load libs" as much as you and I would like, based on bug reports I've seen, patches, and my investigation of Quicklisp libraries. Attempts to tell people to set their env or otherwise configure their OS/install their libraries correctly usually meets with indignant contentiousness. People load a system, get a failure on the foreign library load, locate the define-foreign-library form, then send a request/patch to add an absolute path for their OS. They don't want to be told to set their environment (they may not even know what that is), they want it to work, and they've found a way to make it work for them. My idea is to: (a) have all the libraries work out of the box, and (b) not have to tell people to fix their OS configuration, when it's not even well-defined how to do that (or, at least, I don't know how to do it for their OS -- it's not always LD_LIBRARY_PATH). This takes care of both problems in many cases. Liam On Sun, Mar 30, 2014 at 12:21 PM, Frank Goenninger <frgo@me.com> wrote:
Hi Liam,
as it does not hurt to have multiple directories in *foreign-library-directories* I personally don't care what the entries are ... One of the first things I do is set it to nil .. The OS does know how to load libs. If someone wants to add dirs then please do this via env vars!!
Just my two cents.
Cheers Frank
PS I don't understand all the discussion around this. Proper env var setting just about cures everything.
Von meinem iPhone gesendet
Am 30.03.2014 um 15:41 schrieb Liam Healy <lnp@healy.washington.dc.us>:
From time to time I have seen requests from users to include an absolute path (starting from "/") in systems that use load-foreign-library under a clause for their favorite OS. This makes me nervous, especially when it is a little-used OS, because I have the feeling the requester's configuration is not typical for that OS and I will later get a request to add a different path when the original requester has moved on.
Ideally, there would never have to be any absolute paths; dlopen would always know where to find libraries, because the OS is always configured in a standard way. However, a quick survey of this topic shows that reality falls far short of this ideal, and remedies are not clear. It is well beyond our capabilities to fix the entire world on this matter.
Between fixing the world and fixing every CFFI-using application one by one, there is the compromise of setting default search paths for each OS in CFFI itself, thereby opening all applications to proper functionality out of the box for most OSes. The variable cffi:*foreign-library-directories* seems like the right thing to set. I've looked through all Quicklisp libraries for absolute paths in uses of load-foreign-library, and found these:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
As requests come in to add an absolute path for an application, they can be referred to this mailing list to request the change here, if it is not an application-specific path. Then it is more likely to be properly vetted for general applicability for all users of that OS, and will be available for all libraries. Does this sound like a reasonable way to handle this problem?
Liam _______________________________________________ Cffi-devel mailing list Cffi-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
Could the lisp application be packaged with a config file containing this list? The user could the add additional directories to the load path instead of messing with the OS. On Sunday, March 30, 2014, Liam Healy <lnp@healy.washington.dc.us> wrote:
Thanks Frank. Unfortunately it is just not true that "The OS does know how to load libs" as much as you and I would like, based on bug reports I've seen, patches, and my investigation of Quicklisp libraries.
Attempts to tell people to set their env or otherwise configure their OS/install their libraries correctly usually meets with indignant contentiousness. People load a system, get a failure on the foreign library load, locate the define-foreign-library form, then send a request/patch to add an absolute path for their OS. They don't want to be told to set their environment (they may not even know what that is), they want it to work, and they've found a way to make it work for them. My idea is to: (a) have all the libraries work out of the box, and (b) not have to tell people to fix their OS configuration, when it's not even well-defined how to do that (or, at least, I don't know how to do it for their OS -- it's not always LD_LIBRARY_PATH).
This takes care of both problems in many cases.
Liam
On Sun, Mar 30, 2014 at 12:21 PM, Frank Goenninger <frgo@me.com<javascript:_e(%7B%7D,'cvml','frgo@me.com');>
wrote:
Hi Liam,
as it does not hurt to have multiple directories in *foreign-library-directories* I personally don't care what the entries are ... One of the first things I do is set it to nil .. The OS does know how to load libs. If someone wants to add dirs then please do this via env vars!!
Just my two cents.
Cheers Frank
PS I don't understand all the discussion around this. Proper env var setting just about cures everything.
Von meinem iPhone gesendet
Am 30.03.2014 um 15:41 schrieb Liam Healy <lnp@healy.washington.dc.us<javascript:_e(%7B%7D,'cvml','lnp@healy.washington.dc.us');> :
From time to time I have seen requests from users to include an absolute path (starting from "/") in systems that use load-foreign-library under a clause for their favorite OS. This makes me nervous, especially when it is a little-used OS, because I have the feeling the requester's configuration is not typical for that OS and I will later get a request to add a different path when the original requester has moved on.
Ideally, there would never have to be any absolute paths; dlopen would always know where to find libraries, because the OS is always configured in a standard way. However, a quick survey of this topic shows that reality falls far short of this ideal, and remedies are not clear. It is well beyond our capabilities to fix the entire world on this matter.
Between fixing the world and fixing every CFFI-using application one by one, there is the compromise of setting default search paths for each OS in CFFI itself, thereby opening all applications to proper functionality out of the box for most OSes. The variable cffi:*foreign-library-directories* seems like the right thing to set. I've looked through all Quicklisp libraries for absolute paths in uses of load-foreign-library, and found these:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
As requests come in to add an absolute path for an application, they can be referred to this mailing list to request the change here, if it is not an application-specific path. Then it is more likely to be properly vetted for general applicability for all users of that OS, and will be available for all libraries. Does this sound like a reasonable way to handle this problem?
Liam _______________________________________________ Cffi-devel mailing list Cffi-devel@common-lisp.net<javascript:_e(%7B%7D,'cvml','Cffi-devel@common-lisp.net');> http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
On Sun, 30 Mar 2014 09:41:08 -0400, Liam Healy said:
From time to time I have seen requests from users to include an absolute path (starting from "/") in systems that use load-foreign-library under a clause for their favorite OS. This makes me nervous, especially when it is a little-used OS, because I have the feeling the requester's configuration is not typical for that OS and I will later get a request to add a different path when the original requester has moved on.
Ideally, there would never have to be any absolute paths; dlopen would always know where to find libraries, because the OS is always configured in a standard way. However, a quick survey of this topic shows that reality falls far short of this ideal, and remedies are not clear. It is well beyond our capabilities to fix the entire world on this matter.
Between fixing the world and fixing every CFFI-using application one by one, there is the compromise of setting default search paths for each OS in CFFI itself, thereby opening all applications to proper functionality out of the box for most OSes. The variable cffi:*foreign-library-directories* seems like the right thing to set. I've looked through all Quicklisp libraries for absolute paths in uses of load-foreign-library, and found these:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
As requests come in to add an absolute path for an application, they can be referred to this mailing list to request the change here, if it is not an application-specific path. Then it is more likely to be properly vetted for general applicability for all users of that OS, and will be available for all libraries. Does this sound like a reasonable way to handle this problem?
I think your proposed unix (Linux) clause assumes a Debian-style split between 32-bit and 64-bit. This will be wrong when using a 32-bit Lisp on 64-bit Debian/Ubuntu and also wrong when using a 64-bit Lisp on 64-bit Fedora/RHEL/CentOS. I'm also not sure about the Solaris cases. It should probably be /usr/lib/64 instead of /usr/lib/amd64, though it may also be wrong to add them for a 32-bit Lisp. __Martin
I'm uncertain what you mean, all that #+unix adds is /usr/lib and /usr/local/lib. How is that Debian-specific? Can you suggest better paths for #+unix, or #+linux if you want? I noticed the Solaris "64" and "amd64" inconsistency, but I'm not a user of Solaris so I don't know what's right. Moreover: all those paths come from actual paths used in quicklisp projects. Presumably, if it works for users of those projects, it works for all users of all projects on those platforms. Conversely, if it doesn't work, it needs to be fixed everywhere. This supports the idea of putting it in a central place; it is already getting more visibility than the discussions on project mailing lists. On Mon, Mar 31, 2014 at 1:24 PM, Martin Simmons <martin@lispworks.com>wrote:
On Sun, 30 Mar 2014 09:41:08 -0400, Liam Healy said:
From time to time I have seen requests from users to include an absolute path (starting from "/") in systems that use load-foreign-library under a clause for their favorite OS. This makes me nervous, especially when it is a little-used OS, because I have the feeling the requester's configuration is not typical for that OS and I will later get a request to add a different path when the original requester has moved on.
Ideally, there would never have to be any absolute paths; dlopen would always know where to find libraries, because the OS is always configured in a standard way. However, a quick survey of this topic shows that reality falls far short of this ideal, and remedies are not clear. It is well beyond our capabilities to fix the entire world on this matter.
Between fixing the world and fixing every CFFI-using application one by one, there is the compromise of setting default search paths for each OS in CFFI itself, thereby opening all applications to proper functionality out of the box for most OSes. The variable cffi:*foreign-library-directories* seems like the right thing to set. I've looked through all Quicklisp libraries for absolute paths in uses of load-foreign-library, and found these:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
As requests come in to add an absolute path for an application, they can be referred to this mailing list to request the change here, if it is not an application-specific path. Then it is more likely to be properly vetted for general applicability for all users of that OS, and will be available for all libraries. Does this sound like a reasonable way to handle this problem?
I think your proposed unix (Linux) clause assumes a Debian-style split between 32-bit and 64-bit. This will be wrong when using a 32-bit Lisp on 64-bit Debian/Ubuntu and also wrong when using a 64-bit Lisp on 64-bit Fedora/RHEL/CentOS.
I'm also not sure about the Solaris cases. It should probably be /usr/lib/64 instead of /usr/lib/amd64, though it may also be wrong to add them for a 32-bit Lisp.
__Martin
_______________________________________________ Cffi-devel mailing list Cffi-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
The problem is that there is no general agreement about where to put libraries on Linux and most 64-bit Linux distros support running 32-bit programs using libraries in a different directory from the 64-bit ones. On 32-bit Debian, /usr/lib is likely to contain 32-bit libraries. On 64-bit Debian, /usr/lib is likely to contain 64-bit libraries. On 32-bit and 64-bit Fedora, /usr/lib is likely to contain 32-bit libraries. Many libraries on later Ubuntu distros are in platform-specific directories such as /usr/lib/i386-linux-gnu and /usr/lib/x64_x86-linux-gnu Therefore, the path you need depends on whether you are running a 64-bit Lisp, a 64-bit OS and also on which distro you are running. In particular, /usr/lib might give you the wrong libraries. __Martin
On Sat, 5 Apr 2014 16:48:31 -0400, Liam Healy said:
I'm uncertain what you mean, all that #+unix adds is /usr/lib and /usr/local/lib. How is that Debian-specific? Can you suggest better paths for #+unix, or #+linux if you want? I noticed the Solaris "64" and "amd64" inconsistency, but I'm not a user of Solaris so I don't know what's right.
Moreover: all those paths come from actual paths used in quicklisp projects. Presumably, if it works for users of those projects, it works for all users of all projects on those platforms. Conversely, if it doesn't work, it needs to be fixed everywhere. This supports the idea of putting it in a central place; it is already getting more visibility than the discussions on project mailing lists.
On Mon, Mar 31, 2014 at 1:24 PM, Martin Simmons <martin@lispworks.com>wrote:
> On Sun, 30 Mar 2014 09:41:08 -0400, Liam Healy said:
From time to time I have seen requests from users to include an absolute path (starting from "/") in systems that use load-foreign-library under a clause for their favorite OS. This makes me nervous, especially when it is a little-used OS, because I have the feeling the requester's configuration is not typical for that OS and I will later get a request to add a different path when the original requester has moved on.
Ideally, there would never have to be any absolute paths; dlopen would always know where to find libraries, because the OS is always configured in a standard way. However, a quick survey of this topic shows that reality falls far short of this ideal, and remedies are not clear. It is well beyond our capabilities to fix the entire world on this matter.
Between fixing the world and fixing every CFFI-using application one by one, there is the compromise of setting default search paths for each OS in CFFI itself, thereby opening all applications to proper functionality out of the box for most OSes. The variable cffi:*foreign-library-directories* seems like the right thing to set. I've looked through all Quicklisp libraries for absolute paths in uses of load-foreign-library, and found these:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
As requests come in to add an absolute path for an application, they can be referred to this mailing list to request the change here, if it is not an application-specific path. Then it is more likely to be properly vetted for general applicability for all users of that OS, and will be available for all libraries. Does this sound like a reasonable way to handle this problem?
I think your proposed unix (Linux) clause assumes a Debian-style split between 32-bit and 64-bit. This will be wrong when using a 32-bit Lisp on 64-bit Debian/Ubuntu and also wrong when using a 64-bit Lisp on 64-bit Fedora/RHEL/CentOS.
I'm also not sure about the Solaris cases. It should probably be /usr/lib/64 instead of /usr/lib/amd64, though it may also be wrong to add them for a 32-bit Lisp.
__Martin
_______________________________________________ Cffi-devel mailing list Cffi-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
On Sun, Mar 30, 2014 at 2:41 PM, Liam Healy <lnp@healy.washington.dc.us> wrote:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
I'm sympathetic towards the goal of making things work out-of-the-box, but I'm worried a list like this could backfire and obfuscate some issues. I'd say that, on unix systems, if dlopen() can't find libraries in /usr/lib then something is deeply wrong with that system. Most of the other paths you list look like they should be configured in ld.so.conf or similar. (I suppose we could open an exception on OS X where signed binaries have an empty lookup path for dlopen().) There is another issue. *FOREIGN-LIBRARY-DIRECTORIES* doesn't affect how the dynamic linker will look up library dependencies. So while it might find libfoo in /opt/local/lib the linker will fail to load dependencies that are also in strange places. Perhaps we can fix that, though. I suppose we can set appropriate environment variables to guide the linker. Placing the 64-bit directories on that list unconditionally seems wrong, as Martin pointed out. What do you think Liam? -- Luís Oliveira http://kerno.org/~luis/
I'm totally in favor of having dlopen work correctly everywhere as the ideal solution. However, as I said, that doesn't always happen, and this is an intermediate solution. I hadn't thought about the secondary library problem, that is a good point. I thought though that we were required to list all libraries needed (even dependent ones) explicitly, or is that just for SBCL? Nevertheless, my point stands: if this is a problem for my proposed solution, it is a problem as things are now, which absolute paths defined in projects that use CFFI. Please keep in mind the proposed list is merely observational. They are culled from existing projects, and if wrong, are also wrong for those projects. Experts in the various OSes, please do propose better paths. Liam On Thu, Apr 3, 2014 at 9:25 AM, Luís Oliveira <luismbo@gmail.com> wrote:
On Sun, Mar 30, 2014 at 2:41 PM, Liam Healy <lnp@healy.washington.dc.us> wrote:
Solaris: /lib/64, /usr/lib/amd64, /usr/lib Darwin: /usr/lib, /opt/local/lib, /usr/local/lib Unix: /usr/local/lib, /usr/lib
Therefore I propose to change the definition to:
(defvar *foreign-library-directories* '(#+(or unix darwin solaris) "/usr/lib" #+(or unix darwin) "/usr/local/lib" #+darwin "/opt/local/lib" #+solaris "/lib/64" #+solaris "/usr/lib/amd64") "List onto which user-defined library paths can be pushed.")
I'm sympathetic towards the goal of making things work out-of-the-box, but I'm worried a list like this could backfire and obfuscate some issues.
I'd say that, on unix systems, if dlopen() can't find libraries in /usr/lib then something is deeply wrong with that system. Most of the other paths you list look like they should be configured in ld.so.conf or similar. (I suppose we could open an exception on OS X where signed binaries have an empty lookup path for dlopen().)
There is another issue. *FOREIGN-LIBRARY-DIRECTORIES* doesn't affect how the dynamic linker will look up library dependencies. So while it might find libfoo in /opt/local/lib the linker will fail to load dependencies that are also in strange places. Perhaps we can fix that, though. I suppose we can set appropriate environment variables to guide the linker.
Placing the 64-bit directories on that list unconditionally seems wrong, as Martin pointed out.
What do you think Liam?
-- Luís Oliveira http://kerno.org/~luis/
On Sat, Apr 5, 2014 at 9:50 PM, Liam Healy <lnp@healy.washington.dc.us> wrote:
I hadn't thought about the secondary library problem, that is a good point. I thought though that we were required to list all libraries needed (even dependent ones) explicitly, or is that just for SBCL? Nevertheless, my point stands: if this is a problem for my proposed solution, it is a problem as things are now, which absolute paths defined in projects that use CFFI.
As long as dlopen's search paths are properly setup, I don't think you need to load dependent libraries explicitly in any Lisp. How about providing this search as an interactive restart when loading via plain dlopen() fails? BTW, loading libraries via absolute paths is usually wrong, since it defeats dlopen()'s search capabilities. This is something we should probably warn about in the User Manual. Created an issue to track that: <https://bugs.launchpad.net/cffi/+bug/1303302>. -- Luís Oliveira http://kerno.org/~luis/
participants (5)
-
Frank Goenninger -
Liam Healy -
Luke Crook -
Luís Oliveira -
Martin Simmons