Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[bot] RubyLLM::Judge (typed judgments) not instrumented

オープン
  • 何か問題がありますか？
#228 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
45/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
ruby
領域
ai, observability

調査の方向性

Start with the patchers list in lib/braintrust/contrib/ruby_llm/integration.rb (around line 49), which currently returns only ChatPatcher, and read patcher.rb plus instrumentation/chat.rb and common.rb to see how spans capture model and usage. Add a judge patcher that wraps RubyLLM::Judge.judge, following the same structure. Done when a judge call produces a span with the questions, input, answers, resolved model, and cost, verified by a test alongside the existing ruby_llm tests.

索引モデルが issue の本文から書いたものです。

説明

ruby

<!-- provider-gap-audit: ruby-llm-judge-not-instrumented -->

Summary

ruby_llm 2.1.0 (released 2026-10-08) introduced RubyLLM::Judge / RubyLLM.judge — a typed-question inference API ("judgments") that returns probabilities, choices, and scores over application data. It is a distinct generative execution surface from Chat#complete (different entry point, different request/response shape, its own default model config, its own cost accounting), and is completely uninstrumented.

This is the same category of gap this SDK has already tracked for RubyLLM.embed/.paint/.transcribe (#136) — a top-level RubyLLM inference API outside Chat that the current integration, which only patches Chat#complete/#execute_tool, cannot see.

What is missing

The ruby_llm integration registers exactly one patcher:

# lib/braintrust/contrib/ruby_llm/integration.rb:49
[ChatPatcher]

and lib/braintrust/contrib/ruby_llm/instrumentation/ contains only chat.rb and common.rb. There is no patcher for RubyLLM::Judge.judge / RubyLLM.judge.

A judgment call takes typed questions (predicate/choice/score), optional structured/image input, and returns a Judgment with named answers, probabilities/confidence, model identity, and cost. Expected instrumentation: a span per judge call capturing the declared questions (names/types), the input, the returned answers (values/probabilities/scores), the resolved model, and token usage/cost (via Judgment#cost) — mirroring how ChatPatcher captures model/usage today.

Braintrust docs status

not_found — https://www.braintrust.dev/docs/integrations/sdk-integrations/ruby-llm documents chat completion spans, tool execution spans, token usage, streaming, and multimodal content for the ruby_llm integration. No mention of judgments/RubyLLM::Judge anywhere on the page.

Upstream sources

Local repo files inspected

  • lib/braintrust/contrib/ruby_llm/integration.rb (line 49) — patchers returns only [ChatPatcher]
  • lib/braintrust/contrib/ruby_llm/patcher.rb — only defines ChatPatcher (wraps RubyLLM::Chat#complete/#execute_tool)
  • lib/braintrust/contrib/ruby_llm/instrumentation/ — contains only chat.rb, common.rb; no judge.rb
  • Grep for Judge/judge across lib/ — zero matches outside unrelated eval-scorer terminology in lib/braintrust/eval.rb (Braintrust's own LLM-as-judge scorers, unrelated to RubyLLM::Judge)
  • Appraisals — ruby_llm minimum tested version is 1.8, latest is >= 2.0, so the appraisal matrix already exercises gem versions that ship RubyLLM::Judge
主要言語
Ruby
スター
9
フォーク
10
平均マージ
3日 43分
マージ済み PR（30日）
9

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

braintrustdata/braintrust-sdk-ruby のほかの issue

braintrustdata/braintrust-sdk-ruby の issue をすべて見る

似ている issue

Ruby の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。