Here's a workaround in the meantime (though it's more verbose)... using java.lang.Short.reverseBytes(short s) as the test case[1]:

;; no such method failure (jstatic "reverseBytes" "java.lang.Short" #x7f)

;; this works (jstatic (jmethod (jclass "java.lang.Short") "reverseBytes" (jclass "short")) (jclass "java.lang.Short") #x7f)

This is also true with instance methods...

(require :abcl-contrib) (require :jss)

(defvar *bb* (#"allocate" 'java.nio.ByteBuffer 1024))

;; no such method failure (jcall "putShort" *bb* 20)

;; works (jcall (jmethod (jclass "java.nio.ByteBuffer") "putShort" (jclass "short")) *bb* 20)

;; works (not a short though) (jcall "putInt" *bb* 20)

Hope this helps! I hope to look into this in more detail (potential problem / enhancement opportunity lies in org.armedbear.lisp.Java's method resolution)... a change here would also let the jss syntax also work: (#"putShort" *bb* 20)

[1] https://docs.oracle.com/javase/7/docs/api/java/lang/Short.html#reverseBytes(...)

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, November 27, 2019 2:28 AM, Brad Knotwell bknotwell@yahoo.com wrote:

I'm pretty sure you're right.  I have a MWE below that shows an unintuitive difference.

Thx.

--Brad

(defun void-function ()

(let* ((result (jstatic "square" "Main" 32)))

(format t "in void-function, result of calling square(32): ~a~%" result)))

(defun void-function-short ()

(let* ((result (jstatic "squareshort" "Main" 16)))

(format t "in void-function, result of calling squareshort(16): ~a~%" result)))

(void-function)

(void-function-short)

import org.armedbear.lisp.*;

public class Main

{

public static int square(int a) {

return a * a;

}

public static int squareshort(short a)

{

return a * a;

}

}

$ java -cp ~/abcl_excel_gen/abcl-bin-1.6.0/abcl.jar:. org.armedbear.lisp.Main

Armed Bear Common Lisp 1.6.0

Java 11.0.1 Oracle Corporation

Java HotSpot(TM) 64-Bit Server VM

Low-level initialization completed in 0.295 seconds.

Startup completed in 1.663 seconds.

Type ":help" for a list of available commands.

CL-USER(1): (load "lispfunctions.lisp")

in void-function, result of calling square(32): 1024

Error loading /Users/bknotwel/abcl_excel_gen/bug/lispfunctions.lisp at line 11 (offset 358)

#<THREAD "interpreter" {560A10B3}>: Debugger invoked on condition of type ERROR

no such method

Restarts:

0: TOP-LEVEL Return to top level.

[1] CL-USER(2):

On Tuesday, November 26, 2019, 3:53:11 PM PST, Mark Evenson evenson@panix.com wrote:

...

On Nov 26, 2019, at 18:38, Brad Knotwell bknotwell@yahoo.com wrote:

I've been using ABCL to get access to a Java library and it's worked generally well.  I have run into one problem I haven't been able to workaround.

I need to make a call to a method that takes a short as an argument.  I'm passing in a value--10--that fits in a short but ABCL can't find the method (presumedly due to a type issue).  I've tried using jcoerce but it's not doing what I expect.  What's the right way to do this?

This is probably a bug in how the Bear’s FFI converts the “10” when it attempts to locate your specific Java call site, but without a test the contains the Java code that you are calling into diagnosing exactly what is going in is difficult.

When I get the time (or unless someone beats me to it), I will try to make such a test to figure out what is going on.

We could certainly do with a test suite that combinatorally probes our call site location logic.

-- "A screaming comes across the sky.  It has happened before but there is nothing to compare to it now."

I've looked into this issue at length now and I think I have something worth discussing and potentially trying[0] . The root issue here lies in how ABCL converts its CL representation of integers to Java, as well as its conversion of Java primitives to other Java primitives.

ABCL's common superclass for all objects (LispObject) includes two methods to convert the lisp object to a Java representation: LispObject.javaInstance() and LispObject.javaInstance(Class). LispObject.javaInstance(Class) presumably tries to convert the internal representation to the type represented by the Class passed in. In practice, this seems to be primarily used for Java primitive types. The current implementation of these methods in ABCL's internal Bignum, DoubleFloat, SingleFloat, Fixnum, LispCharacter, and JavaObject, do not seem to have all the required permutations necessary to support all of the possible Java primitive converting operations.

Even for instances where a conversion /is/ implemented (such as Fixnum.javaInstance(Short) where a Short would be returned), the method lookup strategy in Java.findMethod (which boils down to what Java.isAssignable does) doesn't find the method with the Short parameter. Instead it uses normal rules in the Java Language spec and only looks for methods that potentially /widen/ primitives, and disallows any narrowing without an explicit narrowing cast (and in Brad's example, the literal 16 or 32 is represented by Fixnum, which converts to java.lang.Integer). I think the original intention is to have such methods be called using jcoerce, such as:

(jcoerce 32 (jclass "short")) or (jcoerce 32 (jclass "java.lang.Short"))

This fails in the current implementation, I believe due to a bug in JavaObject(Object, Class) constructor. This only does one type of conversion, to java.lang.Byte, if the intended class is Byte and the Object implements Number.

For jcoerce to work with all primitives, all possible primitive operations need to be covered in JavaObject(Object, Class).

I have attached a patch that adds these conversions to JavaObject(Object, Class). However, I've also changed other places where these primitive conversions are done for consistency. My rule of thumb has been that the Java Language Specification rules[1] should be followed, and the /only/ time a /narrowing/ conversion is allowed where information can potentially be lost, is through the JavaObject(Object, Class) constructor (in Lisp land, called using the (java:jcoerce) function).

As such, the patch also changes the implementation classes Bignum, DoubleFloat, SingleFloat, Fixnum, Character, and JavaObject, to support all possible primitive /widening/ conversions. Their implementations of LispObject.javaInstance(Class) continue to signal type errors for conversions that cannot be performend, but also now for all narrowing operations. Previously /some/ of these classes allowed /some/ narrowing conversions to happen. I think a case could be made for each of these LispObject implementations to allow any primitve conversion. The disadvantage is not knowing an information reducing operation potentially occurred (and is counter to the behavior of not finding narrowing methods such as Brad's original example) -- at the very least if all primitive narrowing operations /were/ allowed from Fixnum.javaInstance(Class) for example, a compiler warning should be emitted (though I'm not sure how that would be implemented).

The following patch may break code that relies on the one conversion that has worked in the past (to byte, in the JavaObject(Object, Class) constructor. In fact, this patch breaks swank. To fix your swank, modify your swank's abcl.lisp to include the now required jcoerce call (in the octets-to-jbytes function): the jstatic call should now look like: ... (around line 191) do (java:jstatic (java:jmethod "java.lang.reflect.Array" "setByte" "java.lang.Object" "int" "byte") "java.lang.reflect.Array" bytes i (java:jcoerce byte "byte")))

What are your thoughts? I dislike backwards incompatible changes like this, but I think the impact is probably low due to the fact that method resolution used rules to only find widening conversions, and the fact that short and byte are not as commonly used in method calls.

Previously, (jfield) and (jproperty) uses would also not signal a type error unless LispObject.javaInstance(Class) did not support the conversion (and like mentioned previously, some classes did some primitive narrowing operations). Uses of these fields will now (consistently) with method calls, error on narrowing conversions unless coerced.

Try the patch (but make sure to modify your swank!)...

Now my old example works like the following: (jstatic "reverseBytes" "java.lang.Short" (jcoerce #x7f "short"))

Note also that the patch didn't break any new tests. I ran tests via: (require 'asdf) (load "~/quicklisp/setup.lisp") (asdf:initialize-source-registry `(:source-registry (:directory ,*default-pathname-defaults*) :inherit-configuration)) (asdf:test-system :abcl)

-Mark

[0] See patch attached (against latest master of https://github.com/armedbear/abcl) (3a54c91f3a0cbb71ae8d161acadecf118a65404c)

[1] Widening Primitve Conversion https://docs.oracle.com/javase/specs/jls/se8/html/jls-5.html#jls-5.1.2

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, December 4, 2019 8:37 PM, Mark Evenson evenson@panix.com wrote:

...

On Nov 27, 2019, at 05:53, somewhat-functional-programmer somewhat-functional-programmer@protonmail.com wrote: Here's a workaround in the meantime (though it's more verbose)... using java.lang.Short.reverseBytes(short s) as the test case[1]: ;; no such method failure (jstatic "reverseBytes" "java.lang.Short" #x7f)

[Tests via ABCL-PROVE added.][ABCL-PROVE]

---

"A screaming comes across the sky. It has happened before but there is nothing to compare to it now."

On Dec 13, 2019, at 20:14, somewhat-functional-programmer somewhat-functional-programmer@protonmail.com wrote:

I've looked into this issue at length now and I think I have something worth discussing and potentially trying[0] . The root issue here lies in how ABCL converts its CL representation of integers to Java, as well as its conversion of Java primitives to other Java primitives.

[…]

What are your thoughts? I dislike backwards incompatible changes like this, but I think the impact is probably low due to the fact that method resolution used rules to only find widening conversions, and the fact that short and byte are not as commonly used in method calls.

After much study, reflection and experimentation, I have decided that I didn't like the incompatible change introduced by this patch in requiring Lisp code to explictly JCOERCE arguments to find Java call sites.

Instead, I [managed to solve the original poster's problems of not being able to call Java methods whose parameters where short or byte][1] by modifying the Java.findMethod() machinery to use the actual arguments that will be used in the call for a decision whether to narrow types for calling a given function makes sense for short or int types. I was unable to find other problems with converting primitive types.

[With a little additional elbow grease][2] , I was able to also code a [fix for finding call-sites][3] that take a varargs parameter such as

java.util.Arrays.asList(T... arg)

[1]: https://github.com/armedbear/abcl/pull/144/commits/bfa6e9e44f854ca0c175e73e358d307a7507eeda [2]: https://github.com/armedbear/abcl/pull/144/commits/9fb39d2c0e445bf45739a4f7bb25938ea23f8b5a [3]: https://abcl.org/trac/ticket/259

Given the major holes pointed out by the someone-functional-programmers examination of of the narrowing/widening convention, I am surprised that there are not more Java calls that can't be made with our current FFI machinery, but after scouring our bug reports and trying various methods I can't find any examples. If anyone knows of a case of the Java FFI not finding call sites with these patches, please pipe up.

--

While reviewing this patch, I noticed that since one can't use CL:COERCE to perform a conversion that loses information, as

(coerce 256 '(unsigned-byte 8))

signals some sort of error for all existing Lisp implementations (SBCL, CCL, ECL, CLISP, Allegro, LispWorks), therefore I would argue that something like

(jcoerce 256 "byte")

should also signal an error. Therefore, JCOERCE should be reworked to signal some sort of error if the conversion would lose information. I continue to work through what we would be needed to both incoporate somewhat-functional-programmer's patch, which allows things like

(jcoerce 2.4 "float")

to work, but that would signal TYPE-ERROR conditions when JCOERCE call that lose information are attempted.

My current inclination would be to release the current state as abcl-1.6.1 without incorporating this patch, as I think the need to [patch ABCL-ASDF's inablity to download Maven artifacts][4] is much more pressing than untangling the incomplete inconsistencies in JCOERCE.

[4]: https://mailman.common-lisp.net/pipermail/armedbear-devel/2020-February/0040...

On Feb 10, 2020, at 02:56, somewhat-functional-programmer somewhat-functional-programmer@protonmail.com wrote:

Thank you for reviewing the patch.

[…]

So read on, with our common understanding that I have no objection at all to the reversal of the patch :-)…

Cool.

I like and also thought about looking at the argument values dynamically to help determine a proper call site. It certainly makes sense, especially for literal values, to not require extra coercion or annotation, to find the proper call sites. However, here are a couple of inconsistencies that arise from the current implementation (git master branch 702405fe97a1778f5d3683a6a9cc4294071061e6):

;; the following signals the "no such method" error, ;; consistent with my understanding of the intent of the new patch (jstatic "reverseBytes" "java.lang.Short" 6300000)

;; however, this version does not throw and error, and silently narrows ;; 63000 the value to a short (let ((method (jmethod "java.lang.Short" "reverseBytes" "short"))) (jstatic method nil 6300000)

Confirmed. Mea Maximus Stupidius Culpa in my patch: I didn’t think about/test an explicitly supplied JMETHOD instance.

;; this version has a different error, IllegalArgumentException ;; as the runtime won't automatically convert 32.3 to a short (let ((method (jmethod "java.lang.Short" "reverseBytes" "short"))) (jstatic method nil 32.3))

Yes, this should probably be caught in the find call site machinery, transformed into a Lisp side TYPE_ERROR.

;; this gives the more familiar no such method error (jstatic "reverseBytes" "java.lang.Short" 32.3)

One annoying thing about Java is the "byte" type is signed -- so these value checks will still potentially cause trouble for methods taking a "byte”.

[…]

That brings me to my last question -- What is (java:jcoerce) supposed to do?

I had incorrectly assumed it was much like a Java cast -- in Java one can cast from any primitive type to any other primitive type (with the exception of boolean). One can also cast within the inheritance hierarchy for references etc.

The docstring for jcoerce says: Attempts to coerce OBJECT into a JavaObject of class INTENDED-CLASS. Raises a TYPE-ERROR if no conversion is possible.

Tentatively, I would suggest that JCOERCE tries as hard as possible to DWIM, signalling a TYPE-ERROR (or possibly a subclass) if any coercian would lose information. Then, JCOERCE is defined as a transformation to values whose inverse round-trips. As such, we would extend JCOERCE to cover as many types as possible under this definition, bringing in all Java primtitive types (float, double, char are what we need I think). For completeness, we should cover all descendents of java.lang.Number. But *not* try to cover java.lang.Character. My thinking here is that “char” has a meaning as an unsigned two bytes value, but not allow coercian of java.lang.Character values as it can potentially wrap both int and char. I could be talked out of the last point if someone can defend the interpolation of such values, but I think users would be too confused.

I agree that we should add a JCAST that potentially strips information according to the JLS widening and narrowing rules for all its floating port horrors.

Here are some examples of the inconsistencies:

[…]

Studying and incorporating it into tests. Thanks for bagging parts of the shaggy monster…

—

Currently, I am for fixing JCOERCE for abcl-1.6.1, and pushing JCAST off to either abcl-1.7 or abcl-2.0, whichever I branch first.

yers in CONS, Mark