[FEATURE] Cap how many tool calls run at once

Open
#1,674 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
ruby
Domain
backend

Research direction

Read the Concurrent Tool Execution section of the Tool Execution docs and the ToolConcurrency class referenced by with_tool_options(concurrency: :threads|:fibers); the measured reproduction is 10 calls in one response peaking at 10 concurrent runs. The issue deliberately leaves the choice open between documenting the existing around_tool_call throttle and adding a max_concurrency: option that queues work to a fixed worker pool, so no work should start until a maintainer picks one. Option 2 would also touch the tool_options macro on Agent and need an api.txt entry, with characterization tests for exclusive tools, pauses and per-call result delivery as the bar for done.

Written by the indexing model from the issue text.

Description

enhancement
Scope check
  • This is core LLM communication (not application logic)
  • This benefits most users (not just my use case)
  • This can't be solved in application code with current RubyLLM
  • I read the Contributing Guide
Due diligence
  • I searched existing issues
  • I checked the documentation
What problem does this solve?

With with_tool_options(concurrency: :threads) or :fibers, every call in a response round starts at once: one thread or one Async::Task per call. calls: (#1668) caps how many of a response's calls run, and answers the rest with an error. rounds: caps how many responses one complete runs. Neither limits how many calls run at the same time. A response with 10 calls to a rate-limited API opens 10 requests at once, and calls: 4 doesn't help: it skips 6 of them.

Measured through the public API, with a counter behind a Mutex and a gate that holds each call until all have started (no timing):

calls: nil, 10 calls in one response -> peak running at once: 10, ran: 10
calls: 4, 10 calls in one response -> peak running at once: 4, ran: 4
Proposed solution

Applications can already do this with around_tool_call, which runs in the thread or fiber that runs the call:

slots = Thread::Queue.new
4.times { slots << true }

chat.with_tool_options(concurrency: :threads).around_tool_call do |_call, &run|
  slot = slots.pop
  begin
    run.call
  ensure
    slots << slot
  end
end
semaphore = Async::Semaphore.new(4)
chat.with_tool_options(concurrency: :fibers).around_tool_call { |_call, &run| semaphore.acquire { run.call } }

With a limit of 2 and 10 calls, both peak at 2 and all 10 calls run and get results. The cost is that every call still gets its own thread or task, most of them waiting.

So there are two options, and I'd like you to pick:

  1. Document the pattern in Tool Execution, under Concurrent Tool Execution, and in the RubyLLM skill. No new API. (My recommendation, per "New concepts earn their place".)
  2. A library option, for example with_tool_options(concurrency: :threads, max_concurrency: 4). ToolConcurrency would start at most that many workers per round and feed them from a queue, so it saves the waiting threads too. nil means no limit, values below 1 raise ArgumentError, both modes take it, and it keeps the current rules: exclusive tools still split rounds, a pause still stops the rounds after it, results still arrive as each call finishes, and Rails executor and instrumentation context work as they do now. It would also need the tool_options macro on Agent and an api.txt entry.
Why this belongs in RubyLLM

It may not need to: option 1 is a few lines of application code. Option 2 is only worth it if starting one thread per call is itself the problem, for example with responses of hundreds of calls (the case behind #1668).

Dominant language
Ruby
Stars
4.5k
Forks
537
Avg merge
7h 30m
Merged PRs (30d)
378

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from crmne/ruby_llm

All issues in crmne/ruby_llm

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.