I just tried to install elephant and ran into an annoying problem which could only be resolved by renaming the cffi/uffi-compat/uffi.asd to uffi-compat.asd. The problem seems to be one of identity theft, since elephant uses uffi, and it was picking up the cffi compatibility code (which does not appear to be sufficiently compatible for elephant), instead of what it expected.
I also know that there has been prior discussion about this - I didn't read all of it, but found some on this list by Googling - I wasn't around the list at the time.
Is there a way that this can be resolved? It seems that both libraries should be able to coexist on a system, as there are applications which expect them to be there. It also seems that UFFI has a prior claim to its own name for a system file.
As it stands, everytime I do a clbuild upgrade I am going to have to make sure I fix the link.
Regards, --Jeff
On Fri, Aug 21, 2009 at 4:01 PM, Jeff Cunningham j.k.cunningham@comcast.net wrote:
I just tried to install elephant and ran into an annoying problem which could only be resolved by renaming the cffi/uffi-compat/uffi.asd to uffi-compat.asd. The problem seems to be one of identity theft, since elephant uses uffi, and it was picking up the cffi compatibility code (which does not appear to be sufficiently compatible for elephant), instead of what it expected.
Elephant breaks UFFI's abstractions and assumes SB-ALIEN is being used behind the scenes. That naturally breaks with cffi-uffi-compat.
CFFI's uffi.asd is just a convenient way of using cffi-uffi-compat without editing third-party systems and I didn't expect it to be picked up automatically by tools such as clbuild.
I would gladly rename it. It would work just as well for its original purpose, since as long as you create a uffi.asd symlink, (asdf:find-system :uffi) will pick it up. However, I suspect there might be other code in clbuild that does in fact use cffi-uffi-compat and I wouldn't want to break it. Can anyone confirm if there is such a dependency?
On Fri, Aug 21, 2009 at 10:00 PM, Attila Lendvaiattila.lendvai@gmail.com wrote:
just a note: not everybody is using symlinks to manage asdf systems. with a different name a .asd scanner would miss it...
That's the goal. :-)
Luís Oliveira wrote:
On Fri, Aug 21, 2009 at 4:01 PM, Jeff Cunningham j.k.cunningham@comcast.net wrote:
I just tried to install elephant and ran into an annoying problem which could only be resolved by renaming the cffi/uffi-compat/uffi.asd to uffi-compat.asd. The problem seems to be one of identity theft, since elephant uses uffi, and it was picking up the cffi compatibility code (which does not appear to be sufficiently compatible for elephant), instead of what it expected.
Elephant breaks UFFI's abstractions and assumes SB-ALIEN is being used behind the scenes. That naturally breaks with cffi-uffi-compat.
This looks to me like an Elephant error triggering an obscure asdf configuration error. Ideally, Elephant should be fixed to properly use UFFI, and the configuration error should be hunted down.
I'm unclear about why asdf finds cffi/uffi-compat/uffi.asd before uffi.asd. If this is a stock part of clbuild, then Jeff should report this problem to them; the clbuild team can decide how they want to solve it. If this was caused by a custom asdf recursive-directory-search in Jeff's init files, then he should fix it there (breadth-first instead of depth-first traversal).
CFFI's uffi.asd is just a convenient way of using cffi-uffi-compat without editing third-party systems and I didn't expect it to be picked up automatically by tools such as clbuild.
I would gladly rename it.
I don't think CFFI should change anything. Its perfectly normal for an upgrade/compatibility layer to reuse the original name. For example, my computer has /usr/bin/cc symlinked to gcc; I don't know anyone who uses the original cc.
- Daniel
On Fri, Aug 21, 2009 at 4:01 PM, Jeff Cunninghamj.k.cunningham@comcast.net wrote:
Is there a way that this can be resolved? It seems that both libraries should be able to coexist on a system, as there are applications which expect them to be there. It also seems that UFFI has a prior claim to its own name for a system file.
As Daniel Herring hinted, one way you can solve this problem is to setup asdf:*central-registry* such that UFFI's uffi.asd takes precedence over the clbuild/systems/uffi.asd one. Would that work for you?
-----Original Message----- From: cffi-devel-bounces@common-lisp.net [mailto:cffi-devel-bounces@common-lisp.net] On Behalf Of Luís Oliveira Sent: Friday, August 21, 2009 4:11 PM To: Jeff Cunningham Cc: cffi-devel@common-lisp.net Subject: Re: [cffi-devel] uffi.asd vs uffi-compat.asd
As Daniel Herring hinted, one way you can solve this problem is to setup asdf:*central-registry* such that UFFI's uffi.asd takes precedence over the clbuild/systems/uffi.asd one. Would that work for you?
You are right - it would solve the problem. But it leaves library compatibility in that awful state that requires too much arcane information to be known to write decent auto-setup tools. But don't you think that in the long run Lisp libraries ought to work together without all kinds of special exceptions? Its one of the things people always complain about with Lisp. Just my two cents.
--Jeff
On Fri, Aug 21, 2009 at 11:23 PM, Cunningham, Jeffrey K.JEFFREY.K.CUNNINGHAM@saic.com wrote:
You are right - it would solve the problem. But it leaves library compatibility in that awful state that requires too much arcane information to be known to write decent auto-setup tools. But don't you think that in the long run Lisp libraries ought to work together without all kinds of special exceptions? Its one of the things people always complain about with Lisp. Just my two cents.
If you are interested in the improving the Lisp ecosystem in general, then you should understand why I'm worried about the potential drawbacks of renaming uffi.asd in the context of clbuild.
The most obvious one that UFFI is not part of clbuild's projects. Secondly, libraries such as CLSQL achieve significantly better performance on SBCL using CFFI-UFFI-COMPAT rather than UFFI. Thirdly, CFFI (and therefore CFFI-UFFI-COMPAT) supports more Lisps than UFFI; so if we disable CFFI-UFFI-COMPAT in clbuild, a couple of libraries will stop working on CLISP and others. There might be further drawbacks.
I hope that makes the issue clearer.
-----Original Message----- From: Luís Oliveira [mailto:luismbo@gmail.com] Sent: Friday, August 21, 2009 5:33 PM To: Cunningham, Jeffrey K. Cc: cffi-devel@common-lisp.net; clbuild-devel Subject: Re: [cffi-devel] uffi.asd vs uffi-compat.asd
If you are interested in the improving the Lisp ecosystem in general, then you should understand why I'm worried about the potential drawbacks of renaming uffi.asd in the context of clbuild.
The most obvious one that UFFI is not part of clbuild's projects. Secondly, libraries such as CLSQL achieve significantly better performance on SBCL using CFFI-UFFI-COMPAT rather than UFFI. Thirdly, CFFI (and therefore CFFI-UFFI-COMPAT) supports more Lisps than UFFI; so if we disable CFFI-UFFI-COMPAT in clbuild, a couple of libraries will stop working on CLISP and others. There might be further drawbacks.
I hope that makes the issue clearer.
-- Luís Oliveira
I agree - it makes complete sense to me that having UFFI present should not downgrade or in any way deteriorate the performance of any other package in a system. And yet we have this situation where there are some packages that still depend on UFFI, so - in efficient as it may be - it still needs to able to coexist for awhile until those dependencies have been removed. What do you suggest is the best course to follow? Not for me necessarily, but for the "ecosystem" in general. Because I think CL suffers from these issues not being worked out in a systematic or predictable way.
--Jeff Cunningham
On Sat, Aug 22, 2009 at 2:20 AM, Cunningham, Jeffrey K. JEFFREY.K.CUNNINGHAM@saic.com wrote:
What do you suggest is the best course to follow?
Fixing Elephant, either by porting the whole thing to CFFI -- or (slightly easier) just the SB-ALIEN bits -- seems like the right thing to do.
On Fri, 21 Aug 2009, Cunningham, Jeffrey K. wrote:
From: Luís Oliveira
As Daniel Herring hinted, one way you can solve this problem is to setup asdf:*central-registry* such that UFFI's uffi.asd takes precedence over the clbuild/systems/uffi.asd one. Would that work for you?
You are right - it would solve the problem. But it leaves library compatibility in that awful state that requires too much arcane information to be known to write decent auto-setup tools. But don't you think that in the long run Lisp libraries ought to work together without all kinds of special exceptions? Its one of the things people always complain about with Lisp. Just my two cents.
[It looks like Elephant needs fixing; the rest of this email explains my view on the "dll hell" Jeff describes.]
The situation you describe is common to all library frameworks in all languages I'm aware of. It can be alleviated by versioned libraries; but even versioning falls down when two libraries want to use different versions of a common base lib.
There seem to be two approaches to solving it in C/C++; I think they apply to lisp with only minor modifications. One is to always statically link dependencies; commercial software usually comes with all dependencies included in the installer or the executable itself; Elephant could include uffi in its tarball and specify it directly. This solution is really only good for finished products, and it makes system-wide library updates rather difficult.
The other approach, championed by GNU Autoconf, is to have the installation process probe the system it is being added to. This gives the packager an opportunity to paper over common versioning issues and the chance to fail gracefully. It also provides a human installer a good opportunity to customize the package for local needs.
I think the way forward is to move away from Common Lisp's traditional load-and-compile-and-everything-else-in-one-step approach. The Common Lisp user needs to be able to configure packages before building them, to be able to specify which dependencies to use on a package-by-package basis. The CL user also needs to be able to build and test packages before installing them. And any auto-rebuilding of dependencies is broken except in the simple case where all final products use the same set of dependencies and incremental recompile gives the same result as a full recompile (individually, each is rarely true in CL).
The next step is for people to differentiate between "stock" and "localized" packages. Very few people compile even a small minority of the programs and libraries on their computer. Whether Apple, BSD, Linux, or Microsoft, a distribution maintainer has installed a core set of tools into default locations (e.g /bin and /usr/bin). End-users are then instructed to install custom applications and libraries into separate areas (e.g. /usr/local/bin, ~/local/bin). In this way, system-wide changes can be safely pushed, and only custom stuff risks breakage.
That's enough rambling for now, Daniel
On Fri, Aug 21, 2009 at 19:10, Luís Oliveiraluismbo@gmail.com wrote:
As Daniel Herring hinted, one way you can solve this problem is to setup asdf:*central-registry* such that UFFI's uffi.asd takes precedence over the clbuild/systems/uffi.asd one. Would that work for you?
I use an automated mechanism to setup my asdf:*central-registry* and it would be a pain if I had to change that. I find it offensive if some library decides that it is "better" than something else and makes that something else unusable or harder to use.
In my own private copy cffi, I have renamed CFFI"s uffi.asd to uffi-compat.asd. I'd recommend that this'd be done in the upstream version, too.
-Hans
On Sat, Aug 22, 2009 at 2:36 AM, Hans Hübner hans.huebner@gmail.com wrote:
I use an automated mechanism to setup my asdf:*central-registry* and it would be a pain if I had to change that. I find it offensive if some library decides that it is "better" than something else and makes that something else unusable or harder to use.
Good thing there was never any such intention, otherwise you would be offended! I'm sorry uffi-compat is not useful for you, I guess...
In my own private copy cffi, I have renamed CFFI"s uffi.asd to uffi-compat.asd. I'd recommend that this'd be done in the upstream version, too.
uffi.asd was added by popular demand. (Or maybe it was just *one* user, I don't recall.) I'd be glad to rename it, but as I've mentioned in a previous email, I'm worried that it'd do more harm than good. Suggestions on how to solve the problem are most welcome.
Luís Oliveira wrote:
uffi.asd was added by popular demand. (Or maybe it was just *one* user, I don't recall.) I'd be glad to rename it, but as I've mentioned in a previous email, I'm worried that it'd do more harm than good. Suggestions on how to solve the problem are most welcome.
1. As it stands, even using asdf:*central-registry* is a problem, because there's no way to use both. Since they both have the same name only one can exist with it using the asdf system in any normal fashion. 2. Both UFFI and CFFI have a right to exist. The gcc analogy isn't valid - these aren't compilers that users expect to be substituting one for another. It doesn't matter that the CFFI people don't like the style of code writing that goes on in the ELEPHANT group. They chose to use UFFI for whatever reason and should have the right to expect that some other development group won't break their code via the ASDF package system unless there is a dependency relationship. 3. Since UFFI came up with the name first, I believe most people would agree they have a prior right to the name.
I think you should change the name of your file to *uffi-cffi-compat.asd
*Then, inform your users of the fact, and if they were relying on an implicit use of CFFI in place of UFFI they should change their projects to *explicitly* use CFFI by way of this definition file.
I think it is the right thing to do.
--Jeff
On Sat, Aug 22, 2009 at 3:26 PM, Jeff Cunningham j.k.cunningham@comcast.net wrote:
As it stands, even using asdf:*central-registry* is a problem, because there's no way to use both. Since they both have the same name only one can exist with it using the asdf system in any normal fashion.
That of course is a more general problem that should be solved with something like this: http://paste.lisp.org/display/45282.
I have seen plenty of people taking advantage of the fact that CFFI-UFFI-COMPAT can easily be used as a drop-in replacement for UFFI. I have never seen anyone trying to use CFFI-UFFI-COMPAT and UFFI simultaneously and I strongly suspect you don't actually have that need either.
If for some reason you do need that, it's fairly straightforward to use RENAME-PACKAGE to remove CFFI-UFFI-COMPAT's UFFI nickname.
Both UFFI and CFFI have a right to exist. The gcc analogy isn't valid - these aren't compilers that users expect to be substituting one for another. It doesn't matter that the CFFI people don't like the style of code writing that goes on in the ELEPHANT group.
By "the CFFI people" I presume you mean me, since the other CFFI developers haven't joined the discussion. I never said I didn't their code style.
They chose to use UFFI for whatever reason and should have the right to expect that some other development group won't break their code via the ASDF package system unless there is a dependency relationship.
How are we breaking their code? Just don't use CFFI-UFFI-COMPAT. If I just rename the system, Elephant/clbuild won't work either. That was my first concern.
Since UFFI came up with the name first, I believe most people would agree they have a prior right to the name.
Nobody is arguing that.
I think you should change the name of your file to uffi-cffi-compat.asd
Like I said, I have no problem with that and I agree that it's the best way in principle. However, I have a couple of concerns that I've mentioned in my previous email which you decided to ignore and instead paint this as some personal conflict between the UFFI and CFFI developers. That was unfortunate.
Then, inform your users of the fact, and if they were relying on an implicit use of CFFI in place of UFFI they should change their projects to explicitly use CFFI by way of this definition file.
Not sure if this is what you mean, but some projects do indeed already use cffi-uffi-compat explicitly (e.g., Edi Weitz's CL-GD uses it in order to support CLISP.) but others don't, namely CL-SQL.
Bottom line is, if I rename CFFI's uffi.asd, it'll just break a bunch stuff within clbuild and Elephant still won't work out of the box. I don't see how that could be regarded as an improvement.
I would be glad to work with you to solve the problem, but renaming uffi.asd is clearly not the first step.
Cheers,
Luís Oliveira wrote:
That of course is a more general problem that should be solved with something like this: http://paste.lisp.org/display/45282.
I have seen plenty of people taking advantage of the fact that CFFI-UFFI-COMPAT can easily be used as a drop-in replacement for UFFI. I have never seen anyone trying to use CFFI-UFFI-COMPAT and UFFI simultaneously and I strongly suspect you don't actually have that need either.
If for some reason you do need that, it's fairly straightforward to use RENAME-PACKAGE to remove CFFI-UFFI-COMPAT's UFFI nickname.
You are absolutely right - I don't have that need. And so far as I know, elephant doesn't either, but I haven't dug into their code. I'm just trying to install it to evaluate it. And I have many packages installed already that use the CFFI-UFFI-COMPAT capability.
Both UFFI and CFFI have a right to exist. The gcc analogy isn't valid - these aren't compilers that users expect to be substituting one for another. It doesn't matter that the CFFI people don't like the style of code writing that goes on in the ELEPHANT group.
By "the CFFI people" I presume you mean me, since the other CFFI developers haven't joined the discussion. I never said I didn't their code style.
Sorry, if I offended you - I didn't mean to offend anybody. I am new to this list and don't know who the developers are. I used the term "CFFI people" as a general term to refer to whoever it was on the list who was criticizing the elephant code rather than deal with the problem which I believe was created here by a using the uffi.asd name which was bound to lead to confusion in installations. I think if you asked any randomly 100 selected users _not_ involved in either of these groups, 98 of them would side with me on this.
They chose to use UFFI for whatever reason and should have the right to expect that some other development group won't break their code via the ASDF package system unless there is a dependency relationship.
How are we breaking their code? Just don't use CFFI-UFFI-COMPAT. If I just rename the system, Elephant/clbuild won't work either. That was my first concern.
I don't understand this comment. Please explain. Elephant isn't trying to use it. It depends on UFFI which wants to see uffi.asd in asdf:*central-registry*. But when it goes there, it gets diverted to your version which breaks it.
Yet, I have other codes - many - which use CFFI, so I don't want to remove that. Its a great package. What does that have to do with clbuild?
Since UFFI came up with the name first, I believe most people would agree they have a prior right to the name.
Nobody is arguing that.
I think you should change the name of your file to uffi-cffi-compat.asd
Like I said, I have no problem with that and I agree that it's the best way in principle. However, I have a couple of concerns that I've mentioned in my previous email which you decided to ignore and instead paint this as some personal conflict between the UFFI and CFFI developers. That was unfortunate.
Sorry, I'm not trying to paint a conflict, and certainly nothing of a personal nature. I'm just trying to get this resolved in what I think is the right way. I can't see anyway to resolve that doesn't involve some pain somewhere. I'm trying to take the long view, and in my opinion, the long view should be to improve the utility of Common Lisp, not UFFI or CFFI.
I don't see how you can argue that anyone should be able to set up a package so that when using common automated methods to create links from asd files it effectively masks another group's package is reasonable. As someone pointed out, it can be worked around, but only by excluding one link altogether, which is just as draconian and undesirable. If you would just name it something unique, they could both coexist and anyone could chose what they wanted to use.
Then, inform your users of the fact, and if they were relying on an implicit use of CFFI in place of UFFI they should change their projects to explicitly use CFFI by way of this definition file.
Not sure if this is what you mean, but some projects do indeed already use cffi-uffi-compat explicitly (e.g., Edi Weitz's CL-GD uses it in order to support CLISP.) but others don't, namely CL-SQL.
Bottom line is, if I rename CFFI's uffi.asd, it'll just break a bunch stuff within clbuild and Elephant still won't work out of the box. I don't see how that could be regarded as an improvement.
I would be glad to work with you to solve the problem, but renaming uffi.asd is clearly not the first step.
What, specifically, will break if you rename uffi.asd to cffi-uff-compat.asd? One solution would be to contact cl-sql devs and tell them you are going to change it. Give them a heads up. I'm guessing Edi already saw the potential problem and used the unique name in advance. Can you think of a single other package?
And, elephant works just fine out of the box so far as their claims go - if I rename your compatibility definition file something else.
--Jeff Cunningham
-
Attila Lendvai -
Cunningham, Jeffrey K. -
Daniel Herring -
dherring＠tentpost.com
-
Hans Hübner -
Jeff Cunningham -
Luís Oliveira