EvalPort adapter proposal: portable export/import for ExCyTIn-Bench question sets and eval results

Empieza comprobando los archivos de SecRL indicados—secgym/evaluator.py, secgym/excytin_env.py y el JSON de la pregunta de ejemplo—frente a SPEC.md de EvalPort y la asignación de campos propuesta. Se considera terminado cuando los maintainers confirmen que la asignación y el repositorio de destino son adecuados; la implementación se haría entonces como un paquete independiente en EvalPort, en lugar de un cambio en SecRL.

Summary

I'd like to propose (not submit — just propose, for feedback) an EvalPort adapter for ExCyTIn-Bench: a converter that reads/writes secgym/questions/**/*.json and ExcytinEnv/Evaluator results as EvalPort EvalSuite/ResultSet documents. EvalPort is a small, framework-agnostic JSON format (spec) for making eval datasets and results portable across harnesses (DeepEval, Promptfoo, Inspect AI, and ~30 other adapters so far) instead of every framework inventing its own schema.

This would not touch SecRL's codebase or need a PR here — following the precedent already shipped for other frameworks (e.g. the Opik adapter), it'd live as a standalone secrl-openeval-adapter package in the EvalPort repo, built against your existing public surface. I'm opening this to check whether that's something you'd find useful or would rather I not spend time on, and to get the mapping checked by people who actually know the data — not to ask for engineering time from you.

What I actually read before writing this

I read secgym/evaluator.py, secgym/excytin_env.py, and a sample question file (secgym/questions/o1/v0/test/incident_38_qa_incident_o1-ga_c42.json) rather than going off the README, so the mapping below is against the real shapes:

• A question is a dict with context, question, answer, solution (an ordered list of golden-path strings), plus graph provenance (start_alert, end_alert, start_entities, end_entities, shortest_alert_path).
• Evaluator is the base class; StaticEvaluator.checking() does exact string match against question["answer"] and returns {"reward": 1|0}; LLMEvaluator.checking() first calls check_single_response() (an LLM-judge fuzzy match, binary reward), and if that fails and question["solution"] is a list, falls through to check_solution(), which grades each solution step and computes a cascading discounted partial-credit reward (discount_factor = 0.4, applied from the last-but-one step backward) — this already lands in [0, 1], which happens to match EvalPort's GraderResult.score range exactly.
• ExcytinEnv (a gymnasium.Env) is a multi-turn SQL-query loop against a per-incident MySQL container (step()/reset(), max_steps=15), not a single-shot Q&A call. get_logging() returns {success, steps, reward, success_query_count, total_query_count, question, trajectory} per episode, where trajectory is the full list of {action, observation, reward, done, info} steps.

Proposed mapping

Question → TestCase

Evaluator → Grader

• StaticEvaluator → standard type exact_match.
• LLMEvaluator.check_single_response → standard type llm_judge (reason from check_ans_response).
• LLMEvaluator.check_solution's cascading step credit doesn't fit any of the 11 well-known grader types, and it isn't the same shape as the spec's weighted aggregation strategy either — weighted aggregates scores across multiple graders on one test case, while this is partial credit within a single grader's step sequence. Per the spec's type-openness rule, that'd be a framework-specific type (excytin_step_credit) with params.handler set, so a generic EvalPort runner without the handler skips it (score: null) cleanly instead of misinterpreting it — it would not silently reproduce your 0.4^n discount curve for a runner that doesn't know ExCyTIn.

ExcytinEnv.get_logging() → ResultSet.results[]

• reward → GraderResult.score (already [0,1], no rescaling needed) and passed (reward == 1, matching your own success field).
• trajectory (the per-step SQL queries/observations) → Result.metadata.trajectory, preserved as-is. This is deliberately not claimed as a portable trace: EvalPort explicitly leaves trace/tool-call-log format out of scope (defers to OTel GenAI semantic conventions), so the trajectory would just be an opaque, faithfully-preserved blob for anyone consuming the ResultSet directly — not something a different framework's harness could replay.

Code sketch

# secrl_openeval_adapter/__init__.py  (proposed shape, lives in the EvalPort repo)

def to_openeval(qa_json: list[dict], suite_id: str, attack: str) -> dict:
    """secgym/questions/**/incident_*_qa.json -> EvalPort EvalSuite"""
    graders = [
        {"id": "gr_static_exact", "type": "exact_match", "params": {"trim_whitespace": True}},
        {"id": "gr_llm_judge", "type": "llm_judge",
         "params": {"model": "<configured judge model>", "prompt": "<FUZZY_ANSWER_CHECK_PROMPT>"}},
        {"id": "gr_step_credit", "type": "excytin_step_credit",
         "params": {"handler": "secrl_openeval_adapter.step_credit", "discount_factor": 0.4}},
    ]
    test_cases = [
        {
            "id": f"{attack}_q{i}",
            "input": f"{q['context']}\n{q['question']}" if q.get("context") else q["question"],
            "expected_output": q["answer"],
            "context": [q["context"]] if q.get("context") else [],
            "graders": ["gr_static_exact", "gr_llm_judge", "gr_step_credit"],
            "metadata": {
                "attack": attack,
                "solution": q.get("solution"),
                "start_alert": q.get("start_alert"), "end_alert": q.get("end_alert"),
                "start_entities": q.get("start_entities"), "end_entities": q.get("end_entities"),
                "shortest_alert_path": q.get("shortest_alert_path"),
            },
        }
        for i, q in enumerate(qa_json)
    ]
    return {"$schema": "https://evalport.org/schema/suite.json", "version": "1.0.0",
            "id": suite_id, "graders": graders, "test_cases": test_cases}


def logs_to_openeval(all_logs: list[dict], suite_id: str, run_id: str) -> dict:
    """ExcytinEnv.all_logs (list of get_logging() dicts) -> EvalPort ResultSet"""
    results = []
    for i, log in enumerate(all_logs):
        results.append({
            "test_case_id": f"{log['question'].get('id', f'q{i}')}",
            "actual_output": log["trajectory"][-1].get("info", {}).get("submitted_answer", ""),
            "grader_results": [{
                "grader_id": "gr_step_credit", "type": "excytin_step_credit",
                "score": log["reward"], "passed": log["reward"] == 1,
            }],
            "passed": log["success"],
            "metadata": {
                "steps": log["steps"],
                "success_query_count": log["success_query_count"],
                "total_query_count": log["total_query_count"],
                "trajectory": log["trajectory"],
            },
        })
    return {"$schema": "https://evalport.org/schema/resultset.json", "version": "1.0.0",
            "suite_id": suite_id, "run_id": run_id,
            "started_at": "<from your existing timestamp in the save_file name>",
            "results": results}

Honest gaps, not glossed over

1. Not single-turn. ExcytinEnv is an interactive multi-step SQL environment against a live MySQL container. EvalPort's TestCase/ResultSet model is input→output-shaped; the adapter can faithfully capture the outcome (question, golden answer, submitted answer, reward) but the interactive trajectory only round-trips as an opaque metadata blob, not a replayable trace — by the spec's own admission, trace format is explicitly non-goal #3 (deferred to OTel GenAI conventions), so this isn't a gap the adapter can close, just one worth naming.
2. No turnkey re-execution. Because scoring depends on a per-incident MySQL container SecRL spins up via Docker, a suite converted to EvalPort can't be re-run by a generic EvalPort runner elsewhere — the value here is portability of the dataset and results for cross-benchmark comparison/reporting, not "run ExCyTIn-Bench questions in Promptfoo."
3. Step-credit fidelity. As noted above, the cascading partial-credit reward needs a custom grader type with a handler; a runner without that handler sees score: null, not a wrong score, but it also can't reproduce your curve.
4. I saw the README's 2026-08-26 note that ACESEvals is now "the recommended way to both benchmark and train using RLVR on ExCyTIn-Bench" — I'm not familiar enough with that repo to say whether this adapter should target it instead of / in addition to secgym directly. Flagging that rather than guessing, in case the answer is "talk to that repo, not this one."

Ask

Mainly: is this worth building, and did I get the mapping right? If someone here who actually knows the eval internals can spot-check the TestCase/Grader mapping (especially whether excytin_step_credit is the right call vs. something simpler), I'll build it as a standalone package in the EvalPort repo — no PR or CLA needed on this side, and no ask for review bandwidth here beyond this thread.

— Sahi, independent contributor (not affiliated with this project)