Remote Ollama server in the future?

[AlvaroNovillo]

Hi! I'm trying to use your amazing package to do a custom Ollama build on a remote server, but I saw that it's not configured to run on remote servers by default. Are you planning to add this enhancement in the near future?

Thanks in advance!

[hadley]

If you explain what that means, I can certainly consider it.

[AlvaroNovillo]

I mean to use the package inside a shiny app running in a server, with Ollama installed in a different server. I don’t really know if that’s, possible. In function chat_ollama(), it uses a local Ollama, could I just introduce a base_url from a different server??

[hadley]

Yes.

[AlvaroNovillo]

Hi again!

Sorry for the confusion! I was trying to input inside the turn_list parameter, thats why it didnt work, but it works fine! Thanks

# Cargar la librería ellmer
library(ellmer)

# Definir la dirección IP y puerto del servidor Ollama
ollama_server <- "yuur_server:11434"  # Reemplaza con la IP/puerto correctos

# Configurar el modelo que se quiere usar en Ollama (por ejemplo, llama2)
ollama_model <- "llama3.2"

# Definir el prompt para la consulta
prompt_text <- "¿Cuál es la capital de Francia?"

# Realizar la consulta a Ollama usando la función chat_ollama
chat <- chat_ollama(
  model = ollama_model,         # Nombre del modelo a usar
  base_url = ollama_server,     # Dirección del servidor de Ollama
  echo = "all"
)

# Mostrar la respuesta del modelo
chat$chat(prompt_text)

[cboettig]

@AlvaroNovillo just a side-note here, but if you are hosting models on your own server and want the server to be able to respond to multiple users simultaneously, you might consider using https://docs.vllm.ai/en/latest/ with ellmer's chat_vllm() support.