[FEATURE] Cap how many tool calls run at once
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 22/100
Research direction
Read the Concurrent Tool Execution section of the Tool Execution docs and the ToolConcurrency class referenced by with_tool_options(concurrency: :threads|:fibers); the measured reproduction is 10 calls in one response peaking at 10 concurrent runs. The issue deliberately leaves the choice open between documenting the existing around_tool_call throttle and adding a max_concurrency: option that queues work to a fixed worker pool, so no work should start until a maintainer picks one. Option 2 would also touch the tool_options macro on Agent and need an api.txt entry, with characterization tests for exclusive tools, pauses and per-call result delivery as the bar for done.
Written by the indexing model from the issue text.
Description
Scope check
- This is core LLM communication (not application logic)
- This benefits most users (not just my use case)
- This can't be solved in application code with current RubyLLM
- I read the Contributing Guide
Due diligence
- I searched existing issues
- I checked the documentation
What problem does this solve?
With with_tool_options(concurrency: :threads) or :fibers, every call in a response round starts at once: one thread or one Async::Task per call. calls: (#1668) caps how many of a response's calls run, and answers the rest with an error. rounds: caps how many responses one complete runs. Neither limits how many calls run at the same time. A response with 10 calls to a rate-limited API opens 10 requests at once, and calls: 4 doesn't help: it skips 6 of them.
Measured through the public API, with a counter behind a Mutex and a gate that holds each call until all have started (no timing):
calls: nil, 10 calls in one response -> peak running at once: 10, ran: 10
calls: 4, 10 calls in one response -> peak running at once: 4, ran: 4
Proposed solution
Applications can already do this with around_tool_call, which runs in the thread or fiber that runs the call:
slots = Thread::Queue.new
4.times { slots << true }
chat.with_tool_options(concurrency: :threads).around_tool_call do |_call, &run|
slot = slots.pop
begin
run.call
ensure
slots << slot
end
end
semaphore = Async::Semaphore.new(4)
chat.with_tool_options(concurrency: :fibers).around_tool_call { |_call, &run| semaphore.acquire { run.call } }
With a limit of 2 and 10 calls, both peak at 2 and all 10 calls run and get results. The cost is that every call still gets its own thread or task, most of them waiting.
So there are two options, and I'd like you to pick:
- Document the pattern in Tool Execution, under Concurrent Tool Execution, and in the RubyLLM skill. No new API. (My recommendation, per "New concepts earn their place".)
- A library option, for example
with_tool_options(concurrency: :threads, max_concurrency: 4).ToolConcurrencywould start at most that many workers per round and feed them from a queue, so it saves the waiting threads too.nilmeans no limit, values below 1 raiseArgumentError, both modes take it, and it keeps the current rules: exclusive tools still split rounds, a pause still stops the rounds after it, results still arrive as each call finishes, and Rails executor and instrumentation context work as they do now. It would also need thetool_optionsmacro onAgentand anapi.txtentry.
Why this belongs in RubyLLM
It may not need to: option 1 is a few lines of application code. Option 2 is only worth it if starting one thread per call is itself the problem, for example with responses of hundreds of calls (the case behind #1668).
- Dominant language
- Ruby
- Stars
- 4.5k
- Forks
- 537
- Avg merge
- 7h 30m
- Merged PRs (30d)
- 378
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from crmne/ruby_llm
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 55/100
crmne/ruby_llm#1682 · 1 comment ·
Maintainers usually reply within 1 day
-
[BUG] FileStore::Directory follows links to folders inside its own folderPossibly taken @yorzi claimed this 1 day ago. Openbug
Difficulty 3/5 1-2 days Newbie friendliness 30/100
crmne/ruby_llm#1673 · 1 comment ·
Maintainers usually reply within 1 day
-
[BUG] FileStore::Directory#list returns nothing when a file vanishes mid-listingPossibly taken @yorzi claimed this 1 day ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 32/100
Maintainers usually reply within 1 day
-
Body logging is tied to the logger's debug level, and streaming debug lines ignore the context loggerPossibly taken @josh-sanders claimed this 1 day ago. Openbug
Difficulty 3/5 Half a day Newbie friendliness 55/100
crmne/ruby_llm#1669 · 2 comments ·
Maintainers usually reply within 1 day
Similar issues
-
Bug Ruby
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
getsentry/sentry-ruby#3211 · 1 comment ·
Maintainers usually reply within 2 days
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
internetee/registry#2984 ·
Maintainers usually reply within 6 days
-
interest-zsv
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
hitobito/hitobito_zsv#6 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
ecosyste-ms/issues#862 ·
Maintainers usually reply within 1 day
-
Redis Lua error `@user_script:52: ERR value is not an integer or out of range` when `window_size` is configuredPossibly taken @mimosa claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day