86: Add default limit for tools completions #87

rhys117 · 2025-04-01T13:20:25Z

Resolves #86

Purpose

Introduces a configurable limit on tool completions to prevent infinite loops and excessive API usage. This feature adds protection against scenarios where AI responses might trigger continuous tool calls.

Implementation Details

Added max_tool_llm_calls configuration option (default: 25 calls)
Per-chat override capability through existing contexts
Implemented ToolCallLimitReachedError when limit is exceeded
Added tracking of tool llm calls via number_of_tool_completions counter
Can be overridden for unlimited tool completions with nil

Usage Example

# Global configuration
RubyLLM.configure do |config|
  config.max_tool_llm_calls = 10  # Set default limit
end

Testing

Added RSpec test cases for OpenAI + Claude + Openrouter
Verified both global configuration and per-chat override functionality
Confirmed error raised when limit is reached

Documentation

Added section in tools.md guide explaining the feature
Included examples for both global and per-chat configuration

TODO

Add VCR cassettes for testing with additional providers (I do not have API keys for these providers)

rhys117 · 2025-04-01T13:24:31Z

lib/ruby_llm/chat.rb

@@ -105,6 +107,10 @@ def handle_tool_calls(response, &)
    end

    def execute_tool(tool_call)
+      raise ToolCallsLimitReachedError, "Tool calls limit reached: #{@max_tool_calls}" if max_tool_calls_reached?


It might be worth discussing if this should be handled via the chat instead of raising an unhandled error.

- Otherwise the chat object could not have 'ask' executed on again due to malformed messages

crmne

changing the whole interface to chat is a bit heavy handed. this should be a simple config change.

crmne · 2025-04-23T20:24:15Z

lib/ruby_llm.rb

-    def chat(model: nil, provider: nil)
-      Chat.new(model: model, provider: provider)
+    def chat(model: nil, provider: nil, max_tool_completions: config.max_tool_completions)
+      Chat.new(model: model, provider: provider, max_tool_completions: max_tool_completions)


changing the whole interface to chat is a bit heavy handed. this should be a simple config change.

Thanks for the feedback @crmne. I've adjusted things so the chat interface isn't modified, and instead, a single instance can use an override from the config using with_max_tool_completions.

Please let me know what you think?

@crmne would love an update here ... I'm getting looping tool calls as well, and would like to avoid implementing my own solution.

crmne

Thank you for your work!

Left you some comments + I'm not a big fan of the naming. max_tool_calls is less wordy. Same thing for all the instances of the name, like the error name and the documentation.

crmne · 2025-06-10T11:25:09Z

spec/ruby_llm/chat_tools_spec.rb

+          { title: Faker::Name.name, score: rand(1000) },
+          { title: Faker::Name.name, score: rand(1000) },
+          { title: Faker::Name.name, score: rand(1000) }


I think we can come up with a fake title instead of adding a dependency for these three lines of code.

crmne · 2025-06-10T11:25:51Z

lib/ruby_llm/active_record/acts_as.rb

+      def with_max_tool_completions(...)
+        to_llm.with_max_tool_completions(...)
+        self
+      end
+


We now have configuration Contexts so we don't need the per-chat max tool completions method.

crmne · 2025-06-10T11:26:52Z

lib/ruby_llm/chat.rb

@@ -11,7 +11,7 @@ module RubyLLM
  class Chat # rubocop:disable Metrics/ClassLength
    include Enumerable

-    attr_reader :model, :messages, :tools
+    attr_reader :model, :messages, :tools, :number_of_tool_completions


No need to have it as attr_reader.

crmne · 2025-06-10T11:27:22Z

lib/ruby_llm/chat.rb

+    def with_max_tool_completions(max_tool_completions)
+      @max_tool_completions = max_tool_completions
+      self
+    end
+


We now have configuration Contexts so we don't need the per-chat max tool completions method.

👌 Much nicer for overridding

crmne · 2025-06-10T11:39:26Z

lib/ruby_llm/chat.rb

+      if max_tool_completions_reached?
+        raise ToolCallCompletionsLimitReachedError, "Tool completions limit reached: #{@max_tool_completions}"
+      end
+
+      @number_of_tool_completions += 1


Shouldn't this be at the top of the method? Say max_tool_completions is 0, that would mean that we should process 0 tool completions, right?

Thanks for the review @crmne, you're right, this should have been before the max_tool_completions_reached? check. I've amended this now along with your other feedback.

I haven't manually tested the new changes yet, but the test cases are passing for the providers I have keys for.

Please let me know what you think.

rhys117 · 2025-06-12T12:52:50Z

lib/ruby_llm/chat.rb

+      if context.config
+        @config = context.config
+        @max_tool_llm_calls = @config.max_tool_llm_calls
+      end


Wrapped this in a conditional to ensure that the config was present on the context. This more closely aligns with the assignment in the initialiser (@config = context&.config || RubyLLM.config)|

If I've made a mistake here, please let me know.

rhys117 added 3 commits April 1, 2025 22:54

feat: add limit to tools calls

e7f7a0f

docs: add docs for removing limit

47c8742

docs: Fix docs

190e9de

rhys117 commented Apr 1, 2025

View reviewed changes

rhys117 added 2 commits April 1, 2025 23:30

docs: improve doc - more accurate & organisation

fe837de

bug: reset tools calls for each new 'ask' call

0f7ce26

rhys117 marked this pull request as draft April 1, 2025 13:54

bug: move error so tool result is still added

6b807cf

- Otherwise the chat object could not have 'ask' executed on again due to malformed messages

crmne added the enhancement New feature or request label Apr 2, 2025

rhys117 added 8 commits April 3, 2025 20:53

merge: main

df12db9

chore: rework approach to rely on completions instead of tool calls

3af500d

bug: fix attr_reader declaration to match number_of_tool_completions

39894a8

test: add better example

4718e2d

docs: correct docs after changing naming/strategy

7885c4b

merge: Merge branch 'main' into max-tool-calls

585af65

docs: add default to docs

df26988

chore: rename error to match new naming

19c0cd1

rhys117 marked this pull request as ready for review April 7, 2025 12:15

rhys117 changed the title ~~86: Add default limit for tools calls~~ 86: Add default limit for tools completions Apr 7, 2025

crmne requested changes Apr 23, 2025

View reviewed changes

rhys117 added 7 commits April 24, 2025 21:25

Merge branch 'main' into max-tool-calls

3ad25f4

chore: reorder configuration accessor and add comment

d94d6dd

style: cops - disable / fix

71dba53

test: add spec for configured limit

6455f08

docs: adjust tools docs

5a44bff

merge: main

17b55cc

bug: ensure with_max_tool_completions available through acts_as helpers

d09ce92

rhys117 mentioned this pull request May 9, 2025

Looping Tool Calls #86

Open

crmne requested changes Jun 10, 2025

View reviewed changes

Merge branch 'main' into max-tool-calls

69af539

rhys117 added 9 commits June 11, 2025 23:18

deps: remove faker gem

018cc3b

chore: use existing context instead of with_max_tool_completions

4cb6765

test: adjust spec for context use

6cb8ec3

chore: rename to max_tool_llm_calls

4cdb924

docs: minor doc correction after rename

f8fc8a5

chore: rename error class

731fcc6

test: ensure spec cases for openrouter, openai, anthropic passing

092ff2c

bug: fix +1 issue for llm tool lomts

a0a1fa8

docs: improve tool docs for max tool llm calls

397438e

rhys117 commented Jun 12, 2025

View reviewed changes

rhys117 requested a review from crmne June 12, 2025 12:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

86: Add default limit for tools completions #87

86: Add default limit for tools completions #87

Uh oh!

rhys117 commented Apr 1, 2025 •

edited

Loading

Uh oh!

rhys117 Apr 1, 2025

Uh oh!

crmne left a comment

Uh oh!

crmne Apr 23, 2025

Uh oh!

rhys117 Apr 24, 2025

Uh oh!

bborn Apr 29, 2025

Uh oh!

crmne left a comment

Uh oh!

crmne Jun 10, 2025

Uh oh!

crmne Jun 10, 2025

Uh oh!

crmne Jun 10, 2025

Uh oh!

crmne Jun 10, 2025

Uh oh!

rhys117 Jun 12, 2025

Uh oh!

crmne Jun 10, 2025

Uh oh!

rhys117 Jun 12, 2025

Uh oh!

rhys117 Jun 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

86: Add default limit for tools completions #87

Are you sure you want to change the base?

86: Add default limit for tools completions #87

Uh oh!

Conversation

rhys117 commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Implementation Details

Usage Example

Testing

Documentation

TODO

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crmne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crmne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhys117 Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rhys117 commented Apr 1, 2025 •

edited

Loading

rhys117 Jun 12, 2025 •

edited

Loading