[bot] Anthropic tool_runner agentic loop not instrumented
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 58/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- ruby
- Domain
- observability
Research direction
Start with lib/braintrust/contrib/anthropic/patcher.rb and instrumentation/beta_messages.rb, then inspect the Anthropic SDK's tool_runner methods named in the issue. Compare the nested tool span pattern in lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb. Done means the runner has an overall trace, each BaseTool#call has a child tool span, and all listed execution modes are covered.
Written by the indexing model from the issue text.
Description
Summary
The Anthropic Ruby SDK provides client.beta.messages.tool_runner(...), a beta agentic loop API that automatically executes tools and manages the multi-turn conversation cycle. This surface is not instrumented. The SDK currently instruments beta.messages.create() and beta.messages.stream() (via BetaMessagesPatcher), so individual LLM calls within the runner may produce spans, but the tool executions and overall agentic run are invisible in traces.
What is missing
The tool_runner API (Anthropic::Resources::Beta::Messages#tool_runner) returns a runner object with these execution methods:
each_message { |msg| ... }— iterates through the agentic loop, auto-executing tools viaBaseTool#callbetween iterationsrun_until_finished— runs the full loop and returns all messagesnext_message— step-by-step manual iterationeach_streaming { |event| ... }— streaming variant of the agentic loop
At each iteration, the runner:
- Sends a message to Claude (this call IS traced via existing
BetaMessagesPatcher) - Detects tool_use blocks in Claude's response
- Executes
BaseTool#callon the matching tool (NOT traced) - Sends tool results back and loops
What instrumentation should capture
Parent span for the agentic run (e.g., anthropic.tool_runner):
- Input: initial messages and tool definitions
- Output: final response after all tool loops complete
- Metrics: aggregate token usage across all iterations, total duration
Child spans for each tool execution (e.g., anthropic.tool.{tool_name}):
- Input: tool name + arguments (from Claude's tool_use block)
- Output: tool result (from
BaseTool#callreturn value) - Span attributes:
{type: "tool"}
This pattern is already established in this repo — the RubyLLM integration creates ruby_llm.tool.{tool_name} child spans with braintrust.span_attributes: {type: "tool"} for each tool execution in lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb.
Braintrust docs status
- Braintrust documents "tool" as a first-class span type: "A tool call made by the model — an external API, code execution, database query, etc." (tracing guide)
- Ruby-specific tool_runner instrumentation: not_found
Upstream sources
- Anthropic tool_runner documentation: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-runner
- Anthropic Ruby SDK helpers: https://github.com/anthropics/anthropic-sdk-ruby/blob/main/helpers.md#3-auto-looping-tool-runner-beta
- Anthropic Ruby SDK on RubyGems: https://rubygems.org/gems/anthropic
Local repo files inspected
lib/braintrust/contrib/anthropic/patcher.rb— definesMessagesPatcherandBetaMessagesPatcher; no tool_runner patcherlib/braintrust/contrib/anthropic/instrumentation/beta_messages.rb— wrapscreate()andstream()only; notool_runnerwrapper- Grep for
tool_runner,BaseTool,each_message,run_until_finishedacrosslib/— zero matches lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb— demonstrates the existing pattern for tool execution tracing with nested spans
- Dominant language
- Ruby
- Stars
- 9
- Forks
- 10
- Avg merge
- 22h 10m
- Merged PRs (30d)
- 6
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from braintrustdata/braintrust-sdk-ruby
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
ruby
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
ruby
Difficulty 4/5 3-5 days Newbie friendliness 48/100
All issues in braintrustdata/braintrust-sdk-ruby
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
rubys/roundhouse#444 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
yegor256/rubocop-elegant#92 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
ecosyste-ms/awesome#812 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
forem/forem#23933 · 1 comment ·
Maintainers usually reply within 1 day