Use TornadoVM for faster inferencing

[kevintanhongann]

I suppose this can be improved by utilizing the GPU using TornadoVM.

[mukel]

Yes, it can!
There's some rough prototypes around, but nothing end-to-end.
There's one that offloads some matmuls to GPU: https://github.com/mikepapadim/llama2.tornadovm.java
Another one that uses oneAPI's shared memory to avoid copying, but since inference is memory bound, integrated GPUs don't give much benefit, bar some minor perf. improvement and power savings.

[kevintanhongann]

main...kevintanhongann:llama3.java:main

I don't know if this makes sense. Tried to run it but it seems that it still needs Graalvm related stuff to work.

How to run mine.

jbang -Dtornado.load.api.implementation=uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph -Dtornado.load.runtime.implementation=uk.ac.manchester.tornado.runtime.TornadoCoreRuntime -Dtornado.load.tornado.implementation=uk.ac.manchester.tornado.runtime.common.Tornado -Dtornado.load.annotation.implementation=uk.ac.manchester.tornado.annotation.ASMClassVisitor -Dtornado.load.annotation.parallel=uk.ac.manchester.tornado.api.annotations.Parallel Llama3.java --model Meta-Llama-3-8B-Instruct-Q4_0.gguf --chat

ongoing troubleshooting...

[kevintanhongann]

Pulling @jjfumero into the discussion.

[mikepapadim]

We are currently working on extending llama3 with TornadoVM without support for quantized types on the GPU without the need to convert the gguf model to use floats.