Hi, Luigi,

I'm posting this message to the mailing list for future reference and in the hope that others might find it useful (this ought all to be properly documented anyhow!). I hope you're okay with this.

To provide some context for those just now tuning in, this is a reply to an inquiry about Objective-CL, an Objective-C bridge I have quietly been working on for the last two months, sadly completely unaware of the existence of CL-ObjC (I most probably wouldn't have started my own project had I known of CL-ObjC). Objective-CL is less featureful than CL-ObjC appears to be (for example, it doesn't let you define classes or methods yet). On the other hand, it's portable not only across CL implementations, but also across operating systems, supporting both Mac OS X and GNUstep targets and potentially lots of processor architectures. The license has not yet been decided upon (meanwhile it's GPL'd, because the GPL is the most restrictive license I'd consider using).

Darcs repository: http://matthias.benkard.de/code/objective-cl

Reference manual: http://matthias.benkard.de/objective-cl/documentation

Now, on to the reply itself.

Actually, but i have to study better your code first

I've not studied your code thoroughly yet either, so I can't be sure what I do differently from CL-ObjC, but I'll just try explaining what goes on in my code.

i d like to know (and merge) how you manage:

Okay, here goes...

The most obvoius difference between ObjCL and CL-ObjC is probably that the latter is pure Lisp, while the former is partially implemented in Objective-C for portability. This is very important, because it immediately eliminates a number of problems and makes their solution trivial (more about that later). Furthermore, I can reuse code from PyObjC in this way, so if Apple decides to switch to LLVM or something, Objective-CL can be made compatible with it easily simply by stealing the approprate code from PyObjC. :)

The only Objective-C source files that I have written specifically for Objective-CL are libobjcl.m and libobjcl.h. All the others are from PyObjC, adapted to Objective-CL.

• Method invocation (my code is slow, it calls compile for every invocation, caching the compiled functions for future invocations)

Method invocation is quite simple in principle. First, we convert all arguments into C values and reserve space for the return value, after which we call the method, discard the C values and cons up the result from the return value. This is precisely what LOW-LEVEL-INVOKE does. The actual method call is done by the Objective-C-layer function objcl_invoke_with_types. It's not a complicated function. You can find it in Objective-C/libobjcl.m (like, incidentally, all other functions in the Objective-C layer that are directly called from Lisp and implemented by yours truly).

I'm not sure I understand what exactly you're caching here. I do a bit of caching myself, namely of method signatures and selectors (the latter by using compiler macros). Interestingly, caching selector lookups is a huge performance boost, even though you might think that calling FIND-SELECTOR, which is a wrapper around a fast C function, should probably not take much time. The reason for this is that caching the selector objects avoids MAKE-INSTANCEing them anew for each method call, which takes much longer than the method call itself (around 4 times as long, IIRC).

Actually, there are two different kinds of caching going on here. Method signatures are cached by INVOKE-BY-NAME using a weak hash table, while selectors are loaded into the image at load time by transforming

(INVOKE obj :STRING-WITH-C-STRING "bla" :ENCODING 0)

into roughly

(INVOKE-BY-NAME obj (LOAD-TIME-VALUE (FIND-SELECTOR '(:STRING-WITH-C-STRING :ENCODING))) "bla" 0).

Now, this isn't quite correct, of course, because FIND-SELECTOR may fail. Because of this, the expansion is actually a bit more verbose:

(INVOKE-BY-NAME obj (LOAD-TIME-VALUE (HANDLER-CASE (FIND-SELECTOR '(:STRING-WITH-C-STRING :ENCODING)) (SERIOUS-CONDITION NIL (WARN (MAKE-CONDITION 'SIMPLE-STYLE-WARNING :FORMAT-CONTROL "~S designates an unknown ~ method selector." :FORMAT-ARGUMENTS (LIST '(:STRING-WITH-C-STRING :ENCODING)))) '(:STRING-WITH-C-STRING :ENCODING)))) "bla" 0)

This is done by the compiler macro for INVOKE. INVOKE-BY-NAME itself has a similar compiler macro. Note that a warning is issued by the generated code at load time when a selector can't be found. This is very useful, because it essentially means that typos in method names are caught at load time (which is normally close to compile time).

Note, by the way, that there are two slightly different functions for method calling. There's PRIMITIVE-INVOKE on the one hand and INVOKE-BY-NAME on the other (INVOKE is just a wrapper around the latter). They essentially work the same, but PRIMITIVE-INVOKE converts arguments in an ad-hoc way, while INVOKE-BY-NAME actually examines the method's signature in order to correctly convert arguments to C values. PRIMITIVE-INVOKE is not strictly needed, but it's very convenient to have, because it enables Objective-CL to inquire methods about their signatures, for example, without resorting to functions implemented in Objective-C.

• Struct by value argument passing and returning (i use an hack working on MacOSX x86 i should check also on GNU/Linux)

We do this by using libffi. It's quite simple: What libffi wants is an array of pointers that point to the arguments that it should pass to the foreign function, as well as a type specification which tells it the structure of these arguments. Storing a pointer to a struct from Lisp is trivial. Generating a type specification is non-trivial, but, well, the PyObjC code does it for us. :)

The other way around (structs as return values) is probably similar, but I don't think we actually support it yet. It may just magically work, or it may not. I haven't tested it at all (frankly, I haven't tested passing structs either, but I'm confident that if it doesn't work right now, it's easy to fix).

• Exception management (i dont support it)

This is next to trivial with Objective-CL's architecture. As objcl_invoke_with_types is implemented in Objective-C, it can catch exceptions easily. It then forwards them to the Lisp layer as a return value. (Note that the return value of the method invocation is stored by libffi into a previously allocated cell and _not_ returned as a return value, so we can use the return value in whichever way we like.)

• Condition protocol

There's not much of a protocol... ^^'

By the way, we don't yet support raising Objective-C exceptions from Lisp. I have no idea whether this is hard or not.

• Memory management (i ignored this subject)

Ah, memory management... This is an interesting topic. I use the trivial-garbage library in order to send `release' messages to objects that are collected by the garbage collector on the Lisp side. memory-management.lisp is very short. It's probably easier for you to read it than for me to explain the way it works. :)

A note about the hash tables used there, though: Their sole purpose is caching (again avoiding having to MAKE-INSTANCE every single return value we get from Objective-C via method calls, which would be very slow). If performance is not an issue, they can be removed without changing the behaviour of the system.

I also interested in your performance hacks like the compiler macros.

I hope that I explained them sufficiently above. Do remember, though, that the compiler macros are not merely performance boosters. They also provide a form of compile-time (load-time, actually) checking by issuing warnings when encountering unknown method names.

Note the style warning:

OBJECTIVE-CL> (compile nil '(lambda (x) (invoke x :do 10 :something 20 :weird 42))) ; in: LAMBDA NIL ; (SB-KERNEL:FLOAT-WAIT) ; ; note: deleting unreachable code ; in: LAMBDA (X) ; (MULK.OBJECTIVE-CL:INVOKE MULK.OBJECTIVE-CL::X :DO 10 :SOMETHING 20 :WEIRD 42) ; --> MULK.OBJECTIVE-CL:INVOKE-BY-NAME ; ==> ; (LOAD-TIME-VALUE ; (HANDLER-CASE (MULK.OBJECTIVE-CL:FIND-SELECTOR '(:DO :SOMETHING :WEIRD)) ; (SERIOUS-CONDITION NIL ; (WARN ; (MAKE-CONDITION ' MULK.OBJECTIVE-CL::SIMPLE-STYLE-WARNING ; :FORMAT-CONTROL "~S designates an unknown ~ ; method selector." ; :FORMAT-ARGUMENTS #)) ; '(:DO :SOMETHING :WEIRD)))) ; ; caught STYLE-WARNING: ; (:DO :SOMETHING :WEIRD) designates an unknown method selector. ; ; compilation unit finished ; caught 1 STYLE-WARNING condition ; printed 1 note #<FUNCTION {B862405}> T NIL

Actually in my todo i have other "external" tasks to complete like complete the CLOS interface and to really integrate

This is something I'm interested in, by the way. What does the mapping between CLOS and Objective-C classes look like? Is there support for Objective-C metaclasses? Do Objective-C methods and CLOS generic functions relate to each other in some way?

CL-ObjC/Emacs/Slime into XCode, providing better documentation and examples.

I don't know XCode, but integration with Gorm is a must, yeah. Oh, by the way, have you looked at the JOURNAL file in my repository? I only recently started it. Its purpose is to document some of my decisions regarding the development of Objective-CL.

Despite that i think that a merge with your work looks promising.

What I find most remarkable is that we seem to have done quite a lot of things in a very, _very_ similar way. Even the function/macro/class names are similar or even identical (OBJC-ID and DEFINE-OBJC-STRUCT, for example -- although I never actually implemented the latter).

But don't get me started on CANONICALIZE-<FOO>. Reading that makes my eyes hurt! The correct spelling is CANONIZE (or CANONISE, as I would have written it). :)

(I know, I know. English is an evolving language, and there's some dispute about this particular issue. Still... I like ranting about this kind of stuff.)

Bye, Matthias