Feedback: exporting JudgeResult/AbsoluteScore output as EvalPort ResultSet for portable comparison

Bắt đầu với src/judge.py để xem xét JudgeResult và các trường score của nó, sau đó kiểm tra compare_results.py và entry point được đề xuất src/exporters/evalport.py. Kiểm tra cách một EvalPort extra tùy chọn có thể đóng gói exporter và xác thực một batch bằng validate_suite và validate_result_set mà không thay đổi core pipeline; hoàn tất khi ResultSet được ánh xạ là hợp lệ và tùy chọn.

Hi maintainers — feedback/idea, not a bug report, filed under the "general feedback" path from the README.

Context on me: I maintain EvalPort, an open (Apache 2.0) interchange spec for portable LLM eval test cases, graders, suites, and results, with a Python/TS SDK (evalport-sdk / openeval) and 30+ real, tested framework adapter packages under adapters/. I'm not affiliated with Microsoft — just an outside contributor who read through src/judge.py and thinks this tool's judge output maps cleanly onto EvalPort's Result/ResultSet shape.

Why I'm opening this: This toolkit's anti-bias judge design is genuinely good — dual-ordering pairwise (pairwise_router_first / pairwise_baseline_first) reconciled into a single pairwise_winner, with disagreement collapsed to a tie rather than picked arbitrarily, plus independent 1–5 AbsoluteScore scoring (accuracy, completeness, clarity, helpfulness, and the overall average). That's exactly the kind of structured judge result EvalPort's Result schema was built to carry portably — so a team benchmarking Model Router here could also run the same prompts through DeepEval, Ragas, or Azure AI Evaluation and diff results side by side, without hand-writing glue code each time.

Concretely, what I'm suggesting (not asking anyone to build — happy to draft it myself as a PR if there's interest): a small, optional exporter — e.g. src/exporters/evalport.py — that takes a JudgeResult and emits an EvalPort-shaped result record. Sketch, using the real fields from judge.py:

# src/exporters/evalport.py
from .judge import JudgeResult

def judge_result_to_evalport(result: JudgeResult) -> dict:
    """Map a JudgeResult onto an EvalPort Result record."""
    return {
        "test_case_id": result.prompt_id,
        "grader_results": [
            {
                "grader_id": "pairwise_dual_order",
                "type": "llm_judge",
                "value": result.pairwise_winner,  # "model_router" | "baseline" | "tie"
                "metadata": {
                    "router_first_winner": result.pairwise_router_first.winner if result.pairwise_router_first else None,
                    "baseline_first_winner": result.pairwise_baseline_first.winner if result.pairwise_baseline_first else None,
                },
            },
            {
                "grader_id": "absolute_score_router",
                "type": "llm_judge",
                "value": result.router_score.overall if result.router_score else None,
                "metadata": result.router_score.__dict__ if result.router_score else {},
            },
            {
                "grader_id": "absolute_score_baseline",
                "type": "llm_judge",
                "value": result.baseline_score.overall if result.baseline_score else None,
                "metadata": result.baseline_score.__dict__ if result.baseline_score else {},
            },
        ],
        "metadata": {
            "judge_model": result.judge_model,
            "latency_ms": result.latency_ms,
            "error": result.error,
        },
    }

Wrapping a batch of these plus a suite_id and grader definitions into an EvalPort ResultSet and validating it with evalport-sdk's validate_suite/validate_result_set would be maybe 20-30 more lines, following the shape already used by two adapters in the repo I maintain: azure-ai-evaluation-openeval-adapter (closest analog — same Azure AI ecosystem) and deepeval-openeval-adapter (comparable dual-scoring LLM-judge output).

Benefit for this repo specifically: it'd let compare_results.py-style comparisons extend beyond two local runs to any EvalPort-emitting eval tool, without this project taking on a hard dependency — the export could be fully optional (pip install -e ".[evalport]") and touch nothing in the core judge.py pipeline.

Happy to open a draft PR behind an extras flag if that's useful, or just leave this as food for thought if it's not a direction you want to take the tool. Either way, appreciate the clean dataclass design in judge.py — made this easy to reason about from the outside.

— Sahi, independent contributor (not affiliated with Microsoft)