Author: agallagher Date: Thu Nov 23 20:15:43 2006 New Revision: 3

Added: trunk/notes/sbclproblem/ trunk/notes/sbclproblem/README.txt trunk/notes/sbclproblem/m1.txt trunk/notes/sbclproblem/m2.txt trunk/notes/sbclproblem/m3.txt trunk/notes/sbclproblem/m4.txt trunk/notes/sbclproblem/m5.txt trunk/notes/sbclproblem/m6.txt trunk/notes/sbclproblem/m7.txt trunk/notes/sbclproblem/m8.txt Log: added thread of messages of 2006-06-10, describing sbcl problem

Added: trunk/notes/sbclproblem/README.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/README.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,4 @@ +These messages are from a comp.lang.lisp thread of around 10/06/2006, discussing outstanding problems in python-on-lisp. These problems interfered with sbcl compatibility. + +2006-11-24T01:09 +alexis \ No newline at end of file

Added: trunk/notes/sbclproblem/m1.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m1.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,52 @@ +From: "Alexis Gallagher" public2@alexisgallagher.com +Date: 10 June 2006 06:19:22 BDT +To: gardeners@lispniks.com +Subject: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +Would you consider helping to test just one new library for Common Lisp? +What if the same help would add about a hundred mature and stable +libraries to Common Lisp in one stroke? Yes, I am pitching something +here... + +The Python-On-Lisp gardeners project has been making steady progress, +and is at the point now where it needs some fresh eyes to shake out the +last compatibility issues which prevent if from being brain dead easy to +use. Jeremy Smith started this project a few months ago. Python has many +stable and easy-to-use standard libraries, which some believe CL lacks. +Python has also been designed for easy embedding. Python-on-lisp lets +you access all the Python libraries directly from Common Lisp. Jeremy's +original version worked on Windows in CLisp, Allegro, and LispWorks, but +the code still required some manual configuration to setup. + +Since then, I have worked with Jeremy (and with help from Gary King) to +make python-on-lisp asdf-installable. To use python-on-lisp now, all you +need to do is asdf-install its tarball. This will automatically install +and load CFFI as needed, and find your copy of Python. That's the idea. +But we've reached the point where we need help to fix the last few +issues standing in the way of this nirvana of cross-language +compatibilty. This is what remains: + +1) tweaking the single (CFFI:DEFINE-FOREIGH-LIBRARY ...) form so that it +automatically finds Python for the main Linux distributions. OS X and +Windows seem to work fine, but we don't understand some crucial +subtleties in how Linux stores its shared object libraries. We're aiming +for Debian first, then Ubuntu and Red Hat. + +2) testing and tweaking for compatibility with other CL implementations. +SBCL and CMUCL are presenting problems which are mysterious to us, but +probably transparent to someone familiar with CFFI. We're stuck, but we +think it's a simple issue (a foreign string pointer is being converted +to a CL string prematurely, but we can't figure out why). + +The asdf tarball for python-on-lisp is here: +http://common-lisp.net/project/python-on-lisp/pythononlisp.tar.gz . + +If you've got a minute, please give it a spin. + +cheers, +Alexis & Jeremy +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners

Added: trunk/notes/sbclproblem/m2.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m2.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,72 @@ +From: Stuart Sierra mail@stuartsierra.com +Date: 11 June 2006 03:55:50 BDT +To: Tending the Lisp Garden gardeners@lispniks.com +Subject: Re: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +"Alexis Gallagher" public2@alexisgallagher.com writes: +> The Python-On-Lisp gardeners project has been making +> steady progress, and is at the point now where it needs +> some fresh eyes to shake out the last compatibility issues +> which prevent if from being brain dead easy to use. + +Well, I can happily report that it ran fine on the first +try. (I don't use ASDF-INSTALL, I just downloaded the +tarball and symlinked the .asd file to my central registry.) +Details below. + +Now I should get of my rear and finish Perl-in-Lisp. +Between the two, CL will suddenly be awash in libraries! + +-Stuart + + + +Details of my results, Gentoo Linux running on an AMD64 in +native 64-bit mode: + +$ python +Python 2.4.2 (#1, Oct 29 2005, 15:40:55) +[GCC 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)] on linux2 + +$ find /usr/lib/ -name '*python*.so*' +/usr/lib/libpython2.3.so.1.0 +/usr/lib/libpython2.3.so +/usr/lib/libpython2.4.so.1.0 +/usr/lib/libpython2.4.so + + + +in SBCL 0.9.13 :) +CL-USER> (py::py "print "Hello, World!"") +If you can see this, Python is loaded and working +Hello, World! +NIL +CL-USER> (py::get-web-page "http://www.google.com/") +<html><head><meta ... + + + +in CLISP :) +GNU CLISP 2.35 (2005-08-29) (built 3337446125) (memory 3358963094) +[3]> (py::py "print "Hello, World!"") +sys:1: RuntimeWarning: Python C API version mismatch for +module pol: This Python has API version 1012, module pol has +version 1011. +If you can see this, Python is loaded and working +Hello, World! +NIL + + + +in ECL (from CVS) :( +> (asdf:oos 'asdf:load-op :pythononlisp) +;;; Loading "/home/stuart/.asdf/source/pythononlisp/packages.lisp" +;;; Loading "/home/stuart/.asdf/source/pythononlisp/pythononlisp.lisp" +Unable to load any of the alternatives: + ("libpython2.4.so.1.0" "libpython2.3.so.1.0") +Broken at CFFI::HANDLE-LOAD-FOREIGN-LIBRARY-ERROR. +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners

Added: trunk/notes/sbclproblem/m3.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m3.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,42 @@ +From: Marco Gidde marco.gidde@tiscali.de +Date: 12 June 2006 23:16:32 BDT +To: gardeners@lispniks.com +Subject: Re: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +Hi Alexis, + +"Alexis Gallagher" public2@alexisgallagher.com writes: + +> 2) testing and tweaking for compatibility with other CL +> implementations. +> SBCL and CMUCL are presenting problems which are mysterious to us, but +> probably transparent to someone familiar with CFFI. We're stuck, +> but we +> think it's a simple issue (a foreign string pointer is being converted +> to a CL string prematurely, but we can't figure out why). + +I just made a quick test with SBCL 0.9.12 on x86-64 and had to change +one line in pygetarg from + + (string (cffi:foreign-string-to-lisp (cffi:mem-ref +variable :string))) + +to + + (string (cffi:mem-ref variable :string)) + +Someone else posted that everything worked fine for him with a very +similar configuration, so I *guess*, that it might be a change between +different CFFI versions. I ASDF-INSTALLed it a few weeks ago and it is +version 0.9.1. Maybe cvs CFFI is different in this regard? + + +Regards, + +Marco + +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners

Added: trunk/notes/sbclproblem/m4.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m4.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,37 @@ +From: Lu�s Oliveira luismbo@gmail.com +Date: 13 June 2006 00:24:45 BDT +To: gardeners@lispniks.com +Subject: Re: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +Marco Gidde marco.gidde@tiscali.de +writes: +> I just made a quick test with SBCL 0.9.12 on x86-64 and had to change +> one line in pygetarg from +> +> (string (cffi:foreign-string-to-lisp (cffi:mem-ref +> variable :string))) +> +> to +> +> (string (cffi:mem-ref variable :string)) +> +> Someone else posted that everything worked fine for him with a very +> similar configuration, so I *guess*, that it might be a change between +> different CFFI versions. I ASDF-INSTALLed it a few weeks ago and it is +> version 0.9.1. Maybe cvs CFFI is different in this regard? + +Many (?) months ago, way before 0.9.0 (the first actual release of +CFFI), the behaviour of MEM-REF (and others) changed so that the type +translators are called (in this case, the :STRING type translator). So, +MEM-REF will call FOREIGN-STRING-TO-LISP for you. + +-- +Lu�s Oliveira +luismbo (@) gmail (.) com +http://student.dei.uc.pt/~lmoliv/ + +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners

Added: trunk/notes/sbclproblem/m5.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m5.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,64 @@ +From: Jeremy Smith jeremy@decompiler.org +Date: 13 June 2006 01:25:00 BDT +To: Tending the Lisp Garden gardeners@lispniks.com +Subject: Re: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +Lu�s Oliveira wrote: +> +> Marco Gidde marco.gidde@tiscali.de +> writes: +>> I just made a quick test with SBCL 0.9.12 on x86-64 and had to change +>> one line in pygetarg from +>> +>> (string (cffi:foreign-string-to-lisp (cffi:mem-ref +>> variable :string))) +>> +>> to +>> +>> (string (cffi:mem-ref variable :string)) +>> +>> Someone else posted that everything worked fine for him with a very +>> similar configuration, so I *guess*, that it might be a change +>> between +>> different CFFI versions. I ASDF-INSTALLed it a few weeks ago and +>> it is +>> version 0.9.1. Maybe cvs CFFI is different in this regard? +> +> Many (?) months ago, way before 0.9.0 (the first actual release of +> CFFI), the behaviour of MEM-REF (and others) changed so that the type +> translators are called (in this case, the :STRING type translator). +> So, +> MEM-REF will call FOREIGN-STRING-TO-LISP for you. + +Thanks guys, that's great news. I've implemented the fix so it does it +with the 'old' CFFI (the one I use!) which doesn't automatically do +this, and the new one (which does), by using 'typep' to check for +string-ness. + +I'm not sure if this is really the proper use of 'case' but Lisp allows +'if' to specify any of 2 values, so I think it'll work. + +I'll upload it soon if it works with the newer CFFI. I haven't got CVS +running at common-lisp.net yet, but I'll look into it when I have time. + +(defun pygetarg (variable type) + "Helper function for handling python arguments, used by +gen-pycallback. Should never be used directly from code - use defpyfun +with its arg format" + (case type + (long (cffi:mem-ref variable :long)) + ;If it's decoded by the new version of CFFI + (if (typep (cffi:mem-ref variable :string) 'string) + (string (cffi:mem-ref variable :string)))) + (string (cffi:foreign-string-to-lisp (cffi:mem-ref variable +:string)))) + +Jeremy. +-- +| Jeremy Smith BSc (Hons) +| Chief Scientist, Decompiler Technologies +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners

Added: trunk/notes/sbclproblem/m6.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m6.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,53 @@ +From: Matthew D Swank akopa@charter.net +Date: 13 June 2006 02:49:05 BDT +To: Tending the Lisp Garden gardeners@lispniks.com +Subject: Re: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +Marco Gidde wrote: +> I just made a quick test with SBCL 0.9.12 on x86-64 and had to change +> one line in pygetarg from +> +> (string (cffi:foreign-string-to-lisp (cffi:mem-ref +> variable :string))) +> +> to +> +> (string (cffi:mem-ref variable :string)) +> +> Someone else posted that everything worked fine for him with a very +> similar configuration, so I *guess*, that it might be a change between +> different CFFI versions. I ASDF-INSTALLed it a few weeks ago and it is +> version 0.9.1. Maybe cvs CFFI is different in this regard? +> +> +> Regards, +> +> Marco +> +> _______________________________________________ +> Gardeners mailing list +> Gardeners@lispniks.com +> http://www.lispniks.com/mailman/listinfo/gardeners +> +> +This change was necessary to get pol to work on every platform I tested +it on, and I use the most recent darcs version of cffi. + + sbcl-0.9.13 clisp-2.38 acl80 +WinXP* x x x +Gentoo Linux x86 x x x + +All lisps were tested against the version of python-2.3 that comes with +OpenOffice. + +Do you want this traffic to go to the development list instead? + +Matt + +*All lisps in windows blow up horribly when trying to import the UNO +module, but seem to work with built-in modules O.K. +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners

Added: trunk/notes/sbclproblem/m7.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m7.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,24 @@ +From: Lu�s Oliveira luismbo@gmail.com +Date: 13 June 2006 03:36:51 BDT +To: gardeners@lispniks.com +Subject: Re: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +Jeremy Smith jeremy@decompiler.org writes: +> I'll upload it soon if it works with the newer CFFI. I haven't got CVS +> running at common-lisp.net yet, but I'll look into it when I have +> time. + +We use DARCS, not CVS. In any case, you can download a nightly +snapshot. Just upgrade your copy of CFFI and get rid of that IF in your +code. :-) + +-- +Lu�s Oliveira +luismbo (@) gmail (.) com +http://student.dei.uc.pt/~lmoliv/ + +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners

Added: trunk/notes/sbclproblem/m8.txt ============================================================================== --- (empty file) +++ trunk/notes/sbclproblem/m8.txt Thu Nov 23 20:15:43 2006 @@ -0,0 +1,72 @@ +From: Marco Gidde marco.gidde@tiscali.de +Date: 13 June 2006 10:59:43 BDT +To: gardeners@lispniks.com +Subject: Re: [Gardeners] python-on-lisp needs help +Reply-To: Tending the Lisp Garden gardeners@lispniks.com + +Jeremy Smith jeremy@decompiler.org writes: + +> Lu�s Oliveira wrote: +>> +>> Many (?) months ago, way before 0.9.0 (the first actual release of +>> CFFI), the behaviour of MEM-REF (and others) changed so that the type +>> translators are called (in this case, the :STRING type +>> translator). So, +>> MEM-REF will call FOREIGN-STRING-TO-LISP for you. +> +> Thanks guys, that's great news. I've implemented the fix so it does it +> with the 'old' CFFI (the one I use!) which doesn't automatically do +> this, and the new one (which does), by using 'typep' to check for +> string-ness. +> +> I'm not sure if this is really the proper use of 'case' but Lisp +> allows +> 'if' to specify any of 2 values, so I think it'll work. +> +> I'll upload it soon if it works with the newer CFFI. I haven't got CVS +> running at common-lisp.net yet, but I'll look into it when I have +> time. +> +> (defun pygetarg (variable type) +> "Helper function for handling python arguments, used by +> gen-pycallback. Should never be used directly from code - use defpyfun +> with its arg format" +> (case type +> (long (cffi:mem-ref variable :long)) +> ;If it's decoded by the new version of CFFI +> (if (typep (cffi:mem-ref variable :string) 'string) +> (string (cffi:mem-ref variable :string)))) +> (string (cffi:foreign-string-to-lisp (cffi:mem-ref variable +> :string)))) + +If this works for you it is not obvious (to me) why. If the +indentation indicates two IF branches, there is a parenthesis +mismatch. Try to MACROEXPAND the CASE form to see the resulting +code. I would recommend something like + +(defun pygetarg (variable type) + "Helper function for handling python arguments, used by +gen-pycallback. Should never be used directly from code - use defpyfun +with its arg format" + (case type + (long (cffi:mem-ref variable :long)) + (string (if (typep (cffi:mem-ref variable :string) 'string) + (cffi:mem-ref variable :string) + (cffi:foreign-string-to-lisp (cffi:mem-ref variable :string)))))) + +Also using ECASE instead of CASE might be appropriate. + +As Luis already pointed out switching to a more current version of +CFFI is probably a good idea. And mentioning this dependency on a +specific version on the python-on-lisp home page might also motivate +others to update their ancient CFFI version :-) + + +Regards, + +Marco + +_______________________________________________ +Gardeners mailing list +Gardeners@lispniks.com +http://www.lispniks.com/mailman/listinfo/gardeners