Feature Request: Local LLM Integration for Uninterrupted Project Work

[SlavenDj]

Description:

As a developer working on my project, one of the main challenges I’ve encountered is the limitations of using external language models, especially when I reach usage limits or encounter interruptions. This disrupts my workflow, as I often need to wait. To improve my experience and efficiency, I propose the addition of a feature that allows me to select and use a local LLM that I have installed via Ollama.

Problem:

Currently, using an external LLM service limits my productivity. Whenever I hit a usage limit, my project comes to a halt, causing delays and interruptions. While I would prefer not to subscribe to additional services or plans, I do understand the necessity of keeping costs manageable. Having the ability to use a local LLM would allow me to continue working seamlessly without external limitations, saving me time and frustration.

---

Written by GPT because I'm not very good at writing GitHub issues. xD

[matt-gribben]

This or at least the ability to bring your own API key for OpenAI would be very welcome

[gregnr]

Hey folks, thanks for the suggestion. We like the idea of both opening this up to other LLM providers/models and also BYO-API key. The tricky part with supporting other models is that the current prompts/tool calls were designed around gpt-4o, so we will need to think about how these need to adjust based on other models (or allowing users to adjust prompts).

BYO-API key for OpenAI is more straightforward, though it's worth mentioning - you can always self-host this project which will require you to pass in your own OpenAI API keys anyway. At the moment we don't have a way to turn off the rate limiter (even locally) so we should add logic to allow disabling that.

[gregnr]

Tracking ability to disable rate limiter here: #121

[SlavenDj]

I think ollama has api interface simular to the open ai api so u dont have to change anything i may be wrong
https://ollama.com/blog/openai-compatibility

[gregnr]

Yes this is true, and technically already possible via the OPENAI_API_BASE and OPENAI_MODEL env vars if you self-host (see docs). Again though YMMV using other models with the current prompts. At a minimum, be sure the model you choose supports tool calls.

If you do try this with Ollama, please report back! Interested to hear how well it works.