Working with generic types in Java API calls

[tporeba]

I have a system based on Rhino and I am working on providing support for newest JavaScript by integrating graal.js as alternative engine/language. The system has a Java API that can be called from scripts. I cannot find a way to use Java API methods that expect generic collections of java primitives/wrappers or generic methods of classes typed with wrappers.

For example, lets take:

    // Java
    public ObjectList getObjects(ArrayList<Long> objectIds) {
        for (Long id: objectIds) {
            // do something with id
        }
    }

,

    // JavaScript
    var longList = new java.util.ArrayList();
    longList.add(new java.lang.Long(123456));
    util.getObjects(longList)

or one can used a class derived from generic:

    //Java
    public class LongGeneric extends ArrayList<Long>{}

,

    //JavaScript
    var longList2 = new LongGeneric();
    longList2.add(new java.lang.Long(123456));
    util.getObjects(longList2);

Now, when using these in graal.js (version 19.2.1), I'm running into trouble in both cases, because this code does not pass java.lang.Long as argument to the add() method calls. It actually passes java.lang.Integer, because of eager graal.js conversions. This sooner or later ends with casting problems in my Java API code, for example getObjects() throws:
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long.

In the first example I understand that the script does not know that I intend to use ArrayList<Long>. But still I if I could put a Long somehow into that untyped collection, it would be great. However, I don't see any way to do that.

In the second example the LongGeneric class is explicitly typed with , but still graal.js does not convert java.lang.Integer to java.lang.Long while calling add(). Maybe type erasure prevents graal from detecting the conversion possibility?

Do you have any advice on how to overcome these problems?

Note: this looks similar to discussion from #97 especially this comment. Is there any progress on that longValue() subject?

[tporeba]

I'm looking closer into how numeric values behave when being passed via interop calls. I prepared small testing script:

    var numbers = new java.util.ArrayList();   

    var maxShort = java.lang.Short.MAX_VALUE;
    numbers.add("-----------------------------" + maxShort);
    numbers.add(maxShort);
    numbers.add(new java.lang.Short(maxShort));
    numbers.add(new java.lang.Integer(maxShort))
    numbers.add(new java.lang.Long(maxShort));
    numbers.add(new java.lang.Double(maxShort));
    
    var maxInt = java.lang.Integer.MAX_VALUE;
    numbers.add("-----------------------------" + maxInt);
    numbers.add(maxInt);
    numbers.add(new java.lang.Integer(maxInt));
    numbers.add(new java.lang.Long(maxInt));
    numbers.add(new java.lang.Double(maxInt));
    
    var moreThanInt = java.lang.Integer.MAX_VALUE + 1;
    numbers.add("-----------------------------" + moreThanInt);
    numbers.add(moreThanInt);
    numbers.add(new java.lang.Long(moreThanInt));
    numbers.add(new java.lang.Double(moreThanInt));
    
    var moreThanLong = java.lang.Long.MAX_VALUE + 1;
    numbers.add("-----------------------------" + moreThanLong);
    numbers.add(moreThanLong);
    numbers.add(new java.lang.Double(moreThanLong));

    Packages.test.Util.print(numbers);

Here is the Util class:

    package test;    
    import java.util.ArrayList;
    
    public class Util {    
        public static void print(ArrayList numbers){
            for (Object num: numbers) {
                System.out.println(num + " (" + num.getClass() + ")");
            }
        }
    }

And it prints:

-----------------------------32767 (class java.lang.String)
32767 (class java.lang.Integer)
32767 (class java.lang.Integer)
32767 (class java.lang.Integer)
32767 (class java.lang.Integer)
32767.0 (class java.lang.Double)
-----------------------------2147483647 (class java.lang.String)
2147483647 (class java.lang.Integer)
2147483647 (class java.lang.Integer)
2147483647 (class java.lang.Integer)
2.147483647E9 (class java.lang.Double)
-----------------------------2147483648 (class java.lang.String)
2.147483648E9 (class java.lang.Double)
2147483648 (class java.lang.Long)
2.147483648E9 (class java.lang.Double)
-----------------------------9223372036854776000 (class java.lang.String)
9.223372036854776E18 (class java.lang.Double)
9.223372036854776E18 (class java.lang.Double)

I see some inconsistency here. Numbers greater than MAX_INT are double by default - when used as literals, but I can force them to java.lang.Long by using the constructor. This is not possible for smaller numbers. Is this expected effect?

[woess]

As you've already guessed, the problem is that we lose generic type information due to type erasure. In the LongGeneric case we could try to infer the type from the subclass which we currently don't. This is something we will look into in the future.
Another problem is that in JS we eagerly convert numbers to Integer/Double, which is why the new java.lang.Long trick does not work.

For now, my advice would be to work around the issue on the Java side with specific type signatures, so that our interop can do the right value conversion when calling the method; e.g.:

// utility method approach, works with any List
public static void add(List<Long> list, long value) {
    list.add(value);
}

// subclass approach but override method(s) with the specific type
public static class LongGeneric extends ArrayList<Long> {
    @Override
    public boolean add(Long e) {
        return super.add(e);
    }
}

[tporeba]

Thanks for hints, I checked both. Subclass approach is OK, I will probably use it, where I can.
Utility method approach would be acceptable if I had full control of the scripts, which I don't. I would need to convince my script implementation 'community' to adopt this instead of direct usage of list methods. This might be a problem.

I've also found another hacky way to force construct an ArrayList that contains only Long elements.

    // assignment to array forces Integer -> Long conversion
    var arr = java.lang.reflect.Array.newInstance(java.lang.Long.class, 2);
    arr[0] = 123;
    arr[1] = 124;
 
    // array -> List is done on Java side, so Long-s won't be eagerly converted
    var longList = new java.util.ArrayList();
    java.util.Collections.addAll(longList, arr);
    util.getObjects(longList);

This is similar to your "utility method" approach, just more dirty. So same problem here, I cannot enforce this as an official way of calling methods that expect ArrayLists.

And what about that possibility to obtain Long for bigger numbers via new java.lang.Long() ? Shouldn't it be eagerly converted to Double? If it isn't, then maybe you could allow some way to get Long for smaller numbers too?

Then I could just teach my script writers - "this is how you obtain Long in Graal.js".

[woess]

If we don't have a type hint from the Java method signature, there's currently no guarantee that we'll pass the desired Number type. It's unfortunate that the explicit java.lang.Long conversion does not work. Arguably, if you pass Long from Java to JS and then back to Java, you'd want it to be the same. Of course, as soon as you use it in JS arithmetic, all bets are off.
We'll look into preserving Longs in this case, but I can't say if that will happen anytime soon. We could also provide another way to obtain boxed Java primitives in Graal.js, like Java.to(123, "long").

[tporeba]

Yes, this Java.to(123, "long") would be great.

I found another workaround I can do on Java side, which will be transparent from JavaScript perspective - use abstract java.lang.Number. I could change my methods from accepting ArrayList<Long> to ArrayList<Number> and then, inside Java, make necessary conversion to Long if needed.

So changing API from:

public static void handle(ArrayList<Long> numbers){
    for (Long num : numbers) {
         //use num
    }
}

to this:

public static void handle(ArrayList<Number> numbers){
    for (Number num1 : numbers) {
        Long num = num1.longValue();
        // use num
    }
}

Here erasure works for my benefit, because from JavaScript there is no difference in API. So this would be backwards compatible. Rhino-style code that pass collection of Long would work fine, as well as Graal-style mixed collection of Integer and Double. There is some downside of this approach - sometimes it will result in lossy conversion to long, for example when passing BigDecimal, BigInteger or Double outside of long scope. And of course this can be used only for my own API, doesn't help if you have same problem on some external library.

Thanks for hints and help.
PS I don't know if this issue should be closed or kept open as Java.to(123, "long") enhancement request.

[tporeba]

@woess Another case in this area is that it is impossible to call a method that expects java.lang.Long for numbers that exceed some threshold above which double can store integers without loss of precision.

Given the method:

public void process(Long x){
   //...
}

This works in graal.js: process(9007199254740991).
But this doesn't: process(9007199254740992) and results in

TypeError: invokeMember (process) on JavaClass[com.example.Util] failed due to: UnsupportedTypeException

For some reason this works: process(new java.lang.Long("9007199254740992")).
It seems in this case graal.js does not auto-evaluate Long to Double. But I need to use the constructor that accepts string to get there.

[woess]

@tporeba JS does not have a long type, so 9007199254740992 will be interpreted as a double value. Once the double value is outside the "safe" range, the conversion to long will fail, because double does not have enough precision to accurately represent integers > 9007199254740991.
If you do java.lang.Long.valueOf("9007199254740992") you'll actually get a real Long value. That's a side effect of our language interoperability, just passing through a long should not lose precision. So this works, but as soon as you do any arithmetics, you'll get a double result again.
I think you could use JavaScript's BigInt for this purpose. It should automatically convert to a long if it fits in the long range.

[warrenc5]

Hi, I'm late to the party.

I'm also trying to get a java.lang.Long into a host object that only has Object signature. MyObj. (In my case it's a 3rd party library)

I have a test case trying to work it out https://github.com/warrenc5/graaljs-autobox

I'm using small values < 10000 and the Long type was chosen for historical reasons.

Was there anything I can do? I'm on 22.3.1.

I don't see Java.to ( 5001, 'long' ) - TypeError: 5001 is not an Object