Ollama support

[shouryan01]

Ollama is a very popular backend for running local models with a large library of supported models. It would be great to see ollama support

[FarisHijazi]

Does it expose an openai endpoint? If it does, then support should be simple

[shouryan01]

It doesn't, however can make use of LiteLLM to wrap the endpoint to make it openai compatible

[FarisHijazi]

good idea, will work on it soon

been super busy last few months, I'd appreciate if you could try things out and lemme know if you face any issues, it should be simple like just pointing the middleware to the Ollama/litellm openai port

[Mayorc1978]

It doesn't, however can make use of LiteLLM to wrap the endpoint to make it openai compatible

So it's possible to run ollama in docker, and let's say it exposes the usual localhost:11434 port, then using LiteLLM you can convert that port exposure to a OpenAI endpoint ? Or do you need to run the Ollama model right from LiteLLM?

[shouryan01]

It doesn't, however can make use of LiteLLM to wrap the endpoint to make it openai compatible

So it's possible to run ollama in docker, and let's say it exposes the usual localhost:11434 port, then using LiteLLM you can convert that port exposure to a OpenAI endpoint ? Or do you need to run the Ollama model right from LiteLLM?

Looks like LiteLLM provides a docker image

[FarisHijazi]

I'll try to get on it soon after fixing the authentication