Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[bot] ruby_llm extended thinking not captured: thinking_tokens missing from metrics and thinking content missing from output

未关闭
  • 哪里不对？
#164 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
68/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
冷清
技术栈
ruby
领域
observability-sre, tooling

调研方向

先从 lib/braintrust/contrib/ruby_llm/instrumentation/common.rb 及其 parse_usage_tokens 字段映射开始，然后阅读 lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb 中的 capture_non_streaming_output 和 capture_streaming_output。跟踪 response.thinking 和 thinking chunks 的表示方式，以及 response.to_h 如何提供 token 字段。当 thinking_tokens 出现在 span metrics 中，并且 thinking 内容在非流式和流式响应的 span output 中都出现时，即完成。

由索引模型根据 Issue 内容生成。

描述

Summary

RubyLLM added first-class extended thinking support in v1.10.0 (January 2025) via chat.with_thinking(...). When extended thinking is active, model responses include a thinking content block and a separate thinking_tokens usage bucket. Neither is captured by the current ruby_llm instrumentation: thinking_tokens is absent from span metrics and the thinking content block is absent from span output.

What is missing

1. thinking_tokens not captured in metrics

RubyLLM::Message exposes thinking_tokens (and the alias reasoning_tokens) as a distinct field separate from output_tokens. The current parse_usage_tokens helper maps only four fields and ignores thinking tokens entirely:

# lib/braintrust/contrib/ruby_llm/instrumentation/common.rb
field_map = {
  "input_tokens" => "prompt_tokens",
  "output_tokens" => "completion_tokens",
  "cached_tokens" => "prompt_cached_tokens",
  "cache_creation_tokens" => "prompt_cache_creation_tokens"
  # "thinking_tokens" is not mapped
}

Similarly, capture_non_streaming_output and capture_streaming_output in chat.rb extract a fixed set of token keys from response.to_h and do not include :thinking_tokens.

2. Thinking content block not captured in output

For non-streaming calls, capture_non_streaming_output reads response.content as a plain string. For streaming, capture_streaming_output concatenates chunk.content. Neither reads response.thinking (a Thinking object with .text and .signature) or thinking chunks during streaming. As a result, the thinking/reasoning text is invisible in Braintrust traces even when the user explicitly enables extended thinking.

Usage example showing what is produced but not yet captured
chat = RubyLLM.chat(model: "claude-opus-4-5")
            .with_thinking(effort: :high, budget: 8000)
response = chat.ask("What is 15 * 7, step by step?")

response.thinking.text       # reasoning text — not captured in braintrust.output_json
response.thinking_tokens     # e.g. 312 — not captured in braintrust.metrics
response.output_tokens       # e.g. 28  — already captured

Braintrust docs status

not_found — The Braintrust ruby_llm integration docs (https://www.braintrust.dev/docs/integrations/sdk-integrations/ruby-llm) do not mention extended thinking or thinking_tokens. The Braintrust Anthropic integration docs (https://www.braintrust.dev/docs/integrations/ai-providers/anthropic) capture thinking blocks in output for the direct Anthropic SDK, so there is precedent for this kind of detail. The Braintrust tracing guide (https://www.braintrust.dev/docs/guides/traces) describes capturing full model output.

Upstream sources

Local repo files inspected

  • lib/braintrust/contrib/ruby_llm/instrumentation/common.rb — parse_usage_tokens field_map has no thinking_tokens entry
  • lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb:
    • capture_non_streaming_output (line ~405): reads response.content text only; no response.thinking access
    • capture_streaming_output (line ~367): concatenates chunk.content text only; no thinking chunk accumulation
    • Token extraction (lines ~387–399, ~443–451): hardcodes input_tokens, output_tokens, cached_tokens, cache_creation_tokens; omits thinking_tokens
主要语言
Ruby
星标
9
派生
11
平均合并
2 天 3 小时
30 天内合并 PR
5

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

braintrustdata/braintrust-sdk-ruby 的其他 Issue

查看 braintrustdata/braintrust-sdk-ruby 的全部 Issue

相似的 Issue

更多 Ruby Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。