[bot] RubyLLM::Judge (typed judgments) not instrumented
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 45/100
- Tipo de issue
- Nueva funcionalidad
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- ruby
- Área
- ai, observability
Línea de trabajo
Start with the patchers list in lib/braintrust/contrib/ruby_llm/integration.rb (around line 49), which currently returns only ChatPatcher, and read patcher.rb plus instrumentation/chat.rb and common.rb to see how spans capture model and usage. Add a judge patcher that wraps RubyLLM::Judge.judge, following the same structure. Done when a judge call produces a span with the questions, input, answers, resolved model, and cost, verified by a test alongside the existing ruby_llm tests.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
<!-- provider-gap-audit: ruby-llm-judge-not-instrumented -->
Summary
ruby_llm 2.1.0 (released 2026-10-08) introduced RubyLLM::Judge / RubyLLM.judge — a typed-question inference API ("judgments") that returns probabilities, choices, and scores over application data. It is a distinct generative execution surface from Chat#complete (different entry point, different request/response shape, its own default model config, its own cost accounting), and is completely uninstrumented.
This is the same category of gap this SDK has already tracked for RubyLLM.embed/.paint/.transcribe (#136) — a top-level RubyLLM inference API outside Chat that the current integration, which only patches Chat#complete/#execute_tool, cannot see.
What is missing
The ruby_llm integration registers exactly one patcher:
# lib/braintrust/contrib/ruby_llm/integration.rb:49
[ChatPatcher]
and lib/braintrust/contrib/ruby_llm/instrumentation/ contains only chat.rb and common.rb. There is no patcher for RubyLLM::Judge.judge / RubyLLM.judge.
A judgment call takes typed questions (predicate/choice/score), optional structured/image input, and returns a Judgment with named answers, probabilities/confidence, model identity, and cost. Expected instrumentation: a span per judge call capturing the declared questions (names/types), the input, the returned answers (values/probabilities/scores), the resolved model, and token usage/cost (via Judgment#cost) — mirroring how ChatPatcher captures model/usage today.
Braintrust docs status
not_found — https://www.braintrust.dev/docs/integrations/sdk-integrations/ruby-llm documents chat completion spans, tool execution spans, token usage, streaming, and multimodal content for the ruby_llm integration. No mention of judgments/RubyLLM::Judge anywhere on the page.
Upstream sources
- Official guide: https://rubyllm.com/judgments/
- API reference: https://rubyllm.com/api/RubyLLM/Judge.html and https://rubyllm.com/api/RubyLLM/Judgment.html
- Release notes: https://github.com/crmne/ruby_llm/releases — v2.1.0 (2026-10-08): "
RubyLLM::Judgereturns probabilities, choices, and scores through TypeSafe"
Local repo files inspected
lib/braintrust/contrib/ruby_llm/integration.rb(line 49) —patchersreturns only[ChatPatcher]lib/braintrust/contrib/ruby_llm/patcher.rb— only definesChatPatcher(wrapsRubyLLM::Chat#complete/#execute_tool)lib/braintrust/contrib/ruby_llm/instrumentation/— contains onlychat.rb,common.rb; nojudge.rb- Grep for
Judge/judgeacrosslib/— zero matches outside unrelated eval-scorer terminology inlib/braintrust/eval.rb(Braintrust's own LLM-as-judge scorers, unrelated toRubyLLM::Judge) Appraisals—ruby_llmminimum tested version is 1.8, latest is>= 2.0, so the appraisal matrix already exercises gem versions that shipRubyLLM::Judge
- Lenguaje dominante
- Ruby
- Estrellas
- 9
- Forks
- 11
- Merge medio
- 3 d 43 min
- PR fusionados (30 d)
- 9
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de braintrustdata/braintrust-sdk-ruby
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
braintrustdata/braintrust-sdk-ruby#155 ·
Los mantenedores suelen responder en 1 día
-
[BOT ISSUE] OpenAI Chat Completions missing reasoning model and web search metadata parametersAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
braintrustdata/braintrust-sdk-ruby#143 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
braintrustdata/braintrust-sdk-ruby#137 ·
Los mantenedores suelen responder en 1 día
-
ruby
Dificultad 3/5 Medio día Aptitud para principiantes 62/100
braintrustdata/braintrust-sdk-ruby#229 ·
Los mantenedores suelen responder en 1 día
-
ruby
Dificultad 3/5 1-2 días Aptitud para principiantes 50/100
braintrustdata/braintrust-sdk-ruby#227 ·
Los mantenedores suelen responder en 1 día
Todos los issues de braintrustdata/braintrust-sdk-ruby
Issues similares
-
Kanton Column not HiddenAbiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 61/100
hitobito/hitobito_pfadi_de#164 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
Los mantenedores suelen responder en 7 días
-
Tallied counts an insertion even when the underlying insert raises without changing the dataAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 2 días
-
Windows: packaged_agent_ruby? never matches 'Puppet Labs\Puppet', so the openvox gem is not foundAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
-
Falha nos links do postAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100