[FR] Built-in deterministic "Tier 1" scorers — rule-based structural & safety checks that run before LLM judges

Willingness to contribute

Yes. I can contribute this feature independently.

Proposal Summary

MLflow ships 25+ built-in LLM-based scorers (Correctness, Safety, RetrievalGroundedness, etc.) but zero built-in deterministic code scorers for structural and safety checks — things like schema validity, invariants, policy gates, and hard constraints.

This proposal introduces a set of built-in deterministic "Tier 1" scorers that complement the existing LLM judges. These rule-based checks are:

• Free — no LLM API calls
• Instant — microseconds, not seconds
• Reproducible — identical results every run
• Gating — can short-circuit expensive LLM evaluation when outputs are structurally invalid

Proposed built-in scorers

Scorer	What it checks	Category
ExactMatch	Output matches expected response	Invariant
JsonValidity	Output is valid JSON, optionally matching a schema	Schema validity
RegexMatch	Output matches a regex pattern	Policy gate
ContainsKeywords	Output contains required keywords/phrases	Policy gate
LengthBound	Response length within min/max (chars, words, or tokens)	Hard constraint
IsNotEmpty	Response is non-empty and non-whitespace	Invariant
LatencyThreshold	Trace latency under a maximum threshold	Hard constraint
NumericBound	Numeric output within a value range	Hard constraint

Two-phase roadmap

Phase 1 (this proposal): Ship the 8 built-in code scorers as first-class peers to LLM judges, usable in mlflow.genai.evaluate() today.

Phase 2 (follow-up): Add gate=True support so Tier 1 scorers run before LLM judges — if a gate fails, skip expensive probabilistic evaluation for that row. This modifies the evaluation harness execution order and saves real cost at scale.

Motivation

What is the use case for this feature?

Running fast, deterministic structural and safety checks on LLM outputs — either alongside or before probabilistic LLM-as-a-Judge evaluation. Examples:

• Schema validity: Does the output parse as valid JSON? Does it conform to the expected schema?
• Policy gates: Does the response include a required disclaimer? Does it avoid forbidden patterns?
• Hard constraints: Is the response under the token limit? Is the confidence score within bounds?
• Invariants: Is the response non-empty? Does it match the expected format?

Why is this use case valuable to support for MLflow users in general?

1. Every production evaluation pipeline needs both tiers.

In practice, teams always combine deterministic checks with LLM judges. The deterministic checks catch structural failures (malformed JSON, missing fields, empty responses) while LLM judges assess subjective quality (correctness, tone, safety). Today, MLflow provides built-ins only for the second tier.

2. Tier 1 checks should gate Tier 2 — saving cost.

If JsonValidity fails, there's no point calling openai:/gpt-4 to judge "correctness" on garbage output. Running deterministic checks first and skipping LLM calls on structural failures saves real money. At 1,000 eval rows × 5 LLM judges × $0.01/call, even a 20% structural failure rate saves $100/run.

3. The most common checks are undifferentiated boilerplate.

The @scorer decorator documentation shows exact_match and not_empty as examples users copy-paste. Every team independently reimplements these same patterns, handling edge cases (null outputs, type mismatches, encoding) in different ways. Built-in scorers standardize this.

4. Code scorers currently produce worse results than LLM judges.

User-written @scorer functions typically return bare True/False without rationale. LLM judges automatically generate explanations. Built-in code scorers would return well-structured Feedback objects with informative rationale (e.g., "Output length 2,847 chars exceeds maximum of 2,000"), giving parity in the evaluation results UI.

Why is this use case valuable to support for your project(s) or organization?

We evaluate LLM outputs in regulated domains where deterministic constraint validation is mandatory — outputs must be valid JSON, contain specific fields, fall within numerical bounds, and meet format requirements before any subjective quality assessment begins. Having these as MLflow built-ins would standardize our pipeline and enable the "gate before LLM judge" pattern we need for cost control.

Why is it currently difficult to achieve this use case?

• Zero built-in code scorers exist. mlflow.genai.scorers exports 25+ classes — all BuiltInScorer(Judge) subclasses requiring an LLM.
• BuiltInScorer is LLM-coupled. It extends Judge, which requires instructions (an LLM prompt) and feedback_value_type. Code scorers cannot inherit from this hierarchy.
• No gating mechanism. All scorers run in parallel in the ThreadPoolExecutor. There is no way to say "run these checks first, skip the rest if they fail."
• Copy-paste problem. Every team reimplements exact_match, is_valid_json, is_not_empty from the @scorer docstring examples.

Details

Concrete scenarios where this adds immense merit

Scenario 1: RAG pipeline with structured output An RAG system must return valid JSON with answer, sources, and confidence fields. Before judging answer quality with Correctness():

scorers=[
    JsonValidity(schema={"required": ["answer", "sources", "confidence"]}),  # Tier 1
    NumericBound(min_value=0.0, max_value=1.0, field="confidence"),          # Tier 1
    IsNotEmpty(),                                                             # Tier 1
    Correctness(),                                                            # Tier 2 — skip if Tier 1 fails
]

Scenario 2: Customer-facing chatbot with compliance requirements A financial services chatbot must include a disclaimer and stay under length limits:

scorers=[
    ContainsKeywords(keywords=["not financial advice", "consult a professional"]),  # Policy gate
    LengthBound(max_length=500, unit="words"),                                      # Hard constraint
    RegexMatch(pattern=r"^(?!.*\b(guarantee|promise|certain)\b)"),                   # Safety gate
    Safety(),                                                                        # LLM judge
    Correctness(),                                                                   # LLM judge
]

Scenario 3: Agentic tool-calling with latency SLAs An agent pipeline must respond within 3 seconds and produce non-empty output:

scorers=[
    LatencyThreshold(max_latency_seconds=3.0),  # Tier 1 — from trace
    IsNotEmpty(),                                 # Tier 1
    ToolCallCorrectness(),                        # Tier 2
]

Architecture

Introduce BuiltInCodeScorer(Scorer) — a lightweight base that provides serialization and registration without the LLM-specific requirements of Judge:

class BuiltInCodeScorer(Scorer):
    """Base class for built-in deterministic code scorers."""
    name: str
    required_columns: set[str] = set()
    
    @property
    @abstractmethod
    def criteria(self) -> str:
        """Human-readable description of what this scorer checks."""
    
    @property
    def kind(self) -> ScorerKind:
        return ScorerKind.BUILTIN
    
    def _make_feedback(self, *, value: bool, rationale: str) -> Feedback:
        return Feedback(
            name=self.name,
            value=value,
            rationale=rationale,
            source=AssessmentSource(source_type="CODE", source_id=f"mlflow.scorers.{self.__class__.__name__}"),
        )

Usage

import mlflow
from mlflow.genai.scorers import (
    # Tier 1: Deterministic (instant, free)
    JsonValidity,
    ContainsKeywords,
    LengthBound,
    # Tier 2: LLM-based (nuanced, slower)
    Correctness,
    Safety,
)

results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=my_model,
    scorers=[
        JsonValidity(),
        ContainsKeywords(keywords=["disclaimer"]),
        LengthBound(min_length=50, max_length=2000),
        Correctness(),
        Safety(),
    ],
)

Files to change (Phase 1)

File	Change
mlflow/genai/scorers/builtin_code_scorers.py	[NEW] BuiltInCodeScorer base + 8 scorer implementations
mlflow/genai/scorers/__init__.py	Add lazy imports + __all__ exports
tests/genai/scorers/test_builtin_code_scorers.py	[NEW] Unit tests for all scorers
docs/source/llms/genai/evaluation/index.rst	Documentation section for deterministic scorers

Design decisions

• BuiltInCodeScorer(Scorer) not BuiltInCodeScorer(Judge): Code scorers don't use LLMs and shouldn't carry LLM baggage.
• Separate file: builtin_scorers.py is 3000+ lines of LLM judges. Mixing in deterministic scorers would reduce clarity.
• Reuses existing CODE source type and BUILTIN scorer kind: No schema migration needed.
• No new dependencies: Uses Python stdlib (re, json) only.
• All scorers return Feedback with rationale: Parity with LLM judges in the UI.

What machine learning domain(s) is this feature request about?

• domain/genai: LLMs, Agents, and other GenAI-related use cases

What area(s) of MLflow is this feature request about?

• area/evaluation: MLflow model evaluation features, evaluation metrics, and evaluation workflows