Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[BOT ISSUE] llm.rb gem not instrumented

Open
  • Is something wrong?
#140 0 comments 0 reactions 0 assignees View on GitHub

@clutchski is already working on this.

Since Apr 6, 2026.

  • #142 by @clutchski — open

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
ruby
Domain
observability

Research direction

Start with lib/braintrust/contrib.rb and the existing OpenAI, RubyOpenAI, RubyLLM, and Anthropic integrations, then review llm.rb's chat, completion, tool-calling, embedding, and agent APIs. Check Appraisals and braintrust.gemspec for integration setup; done means llm.rb execution surfaces have dedicated instrumentation with coverage for the listed metadata and provider behaviors.

Written by the indexing model from the issue text.

Description

Summary

The llm.rb gem (RubyGems: llm.rb) is an actively maintained Ruby toolkit for building LLM-powered systems that is not instrumented by this SDK. It is a distinct library from ruby_llm (crmne/ruby_llm), which is already instrumented.

llm.rb has 69K+ downloads, reached v4.11.0 (April 2026), and provides execution-oriented API surfaces across 8 providers.

What is missing

No instrumentation exists for any llm.rb execution surface. Key APIs that should be instrumented:

Chat / Completions
  • LLM::Context#chat — stateful chat with conversation history
  • Streaming responses via structured callbacks
  • Stateless completions
Tool Calling
  • Automatic tool orchestration across providers
  • Concurrent tool execution (threads, fibers, async)
  • Tool execution during streaming
Embeddings
  • LLM::Provider#embed — vector embeddings generation
Agents
  • Reusable, preconfigured assistants with tool auto-execution
Supported Providers

OpenAI, Anthropic, Google, DeepSeek, xAI, zAI, Ollama, LlamaCpp

Braintrust docs status

not_found — No mention of llm.rb or llmrb anywhere in Braintrust documentation at https://www.braintrust.dev/docs. The Braintrust SDK integrations page at https://www.braintrust.dev/docs/integrations/sdk-integrations lists "Ruby LLM" (which refers to the ruby_llm gem, a different library).

Upstream sources

Local repo files inspected

  • lib/braintrust/contrib.rb (lines 195–205) — registers only 4 integrations: OpenAI, RubyOpenAI, RubyLLM, Anthropic. No llm.rb integration.
  • lib/braintrust/contrib/ — no directory or files for llm_rb or llmrb
  • Appraisals — no appraisal scenarios for llm.rb
  • braintrust.gemspec — no mention of llm.rb

Notes

llm.rb has built-in OpenTelemetry support, so users can get basic tracing via Braintrust's OTEL integration. However, dedicated instrumentation would capture richer provider-specific metadata (model parameters, token usage breakdowns, tool execution details, cost tracking) consistent with how the SDK instruments ruby_llm, openai, and anthropic today.

Dominant language
Ruby
Stars
9
Forks
10
Avg merge
22h 10m
Merged PRs (30d)
6

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from braintrustdata/braintrust-sdk-ruby

All issues in braintrustdata/braintrust-sdk-ruby

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.