Tool calling via Ollama

[snscaimito]

Hi!

I've been using OpenAI's models through the Spring AI Starter for OpenAI. Now I want to explore the use of local models.

My understanding is that I can use Ollama as an interface to different models. I've tried DeepSeek but learned that it does not support tools. I've tried llama3.2 and now mistral like so:

    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: mistral

With this code:

      String response = chatClient.prompt()
          .advisors(new PromptChatMemoryAdvisor(chatMemory))
          .tools(new DateTimeTools(), accountingTools)
          .user(prompt)
          .call().content();

I'm getting back:

To find out the current day of the week, use the getCurrentDateTime function:

| Day of the Week |
| -------------- |
| $(getCurrentDateTime().dayOfWeek) |

This is not what I expected. I also see in the logs that it tries to do a lot with JSON. My tools don't provide JSON in particular but things formatted as String using toString(). That lazy approach worked with OpenAI's models.

Some pointers in the right direction would be helpful. Thanks!

[snscaimito]

I did experiment some more and so far found that with llama3.3 it doesn't "like" to call functions/tools. gpt-4o does it flawlessly.

Any suggestion for a local model that is good at calling functions/tools?

[ThomasVitale]

Some Ollama models are configured wrong when it comes to tool calling. In particular, llama3.2 very often returns the tool call request as a JSON object in the assistant message content instead of returning a proper tool call. Instead, I didn't experience any issues using qwen2.5.

[snscaimito]

Noticed that with llama3.2. Thanks for pointing out qwen2.5. So far I think llama3.3 is the thing to use. I want to use a local model.