braintrustdata/autoevals

View on GitHub

AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.

Stars: 1k
Forks: 83
Open beginner issues: 0
Indexed issues: 11

Dominant language: Python
License: MIT
Last GitHub push: Jul 24, 2026
Latest indexed: Sep 20, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

11 open issues indexed

Cross-trial consistency (kappa, self-agreement) falls outside the Scorer shape Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

braintrustdata/autoevals#219 · Opened Sep 7, 2026
OrcaRouter provider support for Autoevals Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

braintrustdata/autoevals#218 · Opened Sep 4, 2026
`closed_q_a.yaml` drops OpenAI’s scoping clause; `factuality.yaml` has a traceable typo Open

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

braintrustdata/autoevals#217 · Opened Aug 31, 2026
Proposal: EvalPort adapter for autoevals' Scorer/Score primitives Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

braintrustdata/autoevals#215 · Opened Aug 29, 2026
Package blocks install with npm when using .npmrc file Open

Difficulty 2/5 1-3 hours Newbie friendliness 58/100

braintrustdata/autoevals#212 · Opened Aug 5, 2026
`EmbeddingSimilarity` class-level embedding cache ignores model and prefix — cross-model cache poisoning Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

braintrustdata/autoevals#211 · Opened Aug 4, 2026
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

braintrustdata/autoevals#206 · 1 comment · Opened Jul 22, 2026
isWrapped crashes on custom OpenAI-compatible clients when the braintrust SDK is loaded (TypeError: reading 'completions') Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

braintrustdata/autoevals#205 · Opened Jul 16, 2026
Is the returned ExactMatch Score object the right minimal external-consumer seam? Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

braintrustdata/autoevals#187 · 2 comments · Opened Apr 24, 2026
ContextPrecision does not meet the specification Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

braintrustdata/autoevals#176 · 2 comments · Opened Feb 25, 2026
AnswerRelevancy uses model parameter instead of embedding_model for Embedding call Open

Difficulty 2/5 1-3 hours Newbie friendliness 25/100

braintrustdata/autoevals#166 · Opened Jan 15, 2026

braintrustdata/autoevals

Get new issues in your inbox