feat(engine): optional runtime measurements reported by custom engines

Aperta
#422 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go
Ambito
api, backend

Direzione di ricerca

Start with docs/design/custom-engine.md and locate the custom engine SessionResult decoding and result persistence paths for the local and http transports. The issue defines validation, masking, diagnostics, size limits, and multi-turn requirements, so use its slice 1 acceptance criteria to map the work and identify the relevant tests. Done means valid measurements and diagnostics persist without changing verdicts or outputs, and the three named guides document the #296 sweep.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

A custom engine can report a few fixed numbers about its run in SessionResult: duration_ms, turns, input_tokens and output_tokens. It cannot report any other runtime measurement with a name and a unit, such as decode throughput at given context lengths, time to first token, or effective context length.

Today the only way to carry such numbers is to put a file in artifacts.files. skill-up then archives the file without reading it. Nothing is validated, and nothing reaches result.json, report.json or a later comparison.

The motivating case is #296: long-context decode-speed decay. An instrumented agent or harness, wrapped as a custom engine, measures:

  • decode TPS per request at fixed context steps (8k, 16k, 32k, …);
  • derived scalars: the baseline speed and the half-speed point n½, the context length where decode speed falls to half of baseline.

As agreed in #296, the measurement and its fitting stay in the external implementation. skill-up only needs to receive, validate, persist and show the results.

The discussion in #317 settled the boundary:

  • These are measurements of what happened during execution, not quality scores.
  • They stay separate from the [0,1] named-metric contract of #347.
  • They do not affect PASS/FAIL.
  • They are not aggregated with quality scores by default.
  • A measurement can feed a score only through a scoring rule that an evaluation defines explicitly.

This issue proposes that separate contract.

Proposal

Add an optional measurements object to the custom engine SessionResult, for both the local and http transports. The first slice validates and persists it. Showing it in the HTML/Markdown reports and comparing it across runs come later; see Slices.

{
  "exit_code": 0,
  "final_message": "Decay sweep finished.",
  "measurements": {
    "schema_version": 1,
    "provenance": {
      "producer": "dsh-eval-harness",
      "producer_version": "e90c255",
      "measured_at": "2026-11-17T08:00:00Z",
      "subject": "provider=example model=example-model (the deployment as measured, not the architecture)",
      "method": "Streaming requests at nominal context steps; a per-request nonce defeats prefix caching; temperature 0, max_tokens 256; median of 3 repeats per step."
    },
    "aggregate": [
      {"id": "baseline_decode_tps", "unit": "tok/s", "value": 200.0,
       "description": "1/a of the least-squares fit tps(n) = 1/(a + b*n)"},
      {"id": "half_speed_context", "unit": "token", "value": 67000,
       "description": "context length where the fitted speed is half of baseline"}
    ],
    "per_step": [
      {
        "id": "decode_tps",
        "unit": "tok/s",
        "axis": {"id": "context_tokens", "unit": "token"},
        "points": [
          {"at": 8192, "value": 178.2},
          {"at": 16384, "value": 160.7},
          {"at": 32768, "value": 134.3},
          {"at": 65536, "value": 101.1},
          {"at": 131072, "value": 67.7},
          {"at": 262144, "availability": "MISSING", "reason": "request_timed_out"}
        ]
      }
    ]
  }
}

The five points raised in #317 are covered below.

1. Names and units
  • Names. Every entry has an id matching the judge-id pattern, ^[a-z][a-z0-9_-]{0,63}$.
    • An id is unique within one measurements object, across aggregate and per_step together.
    • Measurement ids live in their own namespace. They never collide with #347 metric ids, and reports label the two kinds separately.
    • The term "metric" stays reserved for #347 scores. Harbor uses "metrics" more broadly: rewards aggregated across trials, plus the operational numbers in ATIF.
  • Units. unit is required. It is an opaque, case-sensitive label of 1–32 characters from [A-Za-z0-9/%._^*-], compared exactly.
    • Recommended spellings: s, ms, token, tok/s, byte, %, usd, and 1 for dimensionless values.
    • skill-up never converts units. The same id with a different unit is a different measurement.
  • Axis. A per_step entry declares what a step is measured along, with its own id and unit:
    • {"id": "context_tokens", "unit": "token"} for a context sweep;
    • {"id": "step", "unit": "1"} for agent steps;
    • {"id": "elapsed", "unit": "s"} for a time series.
2. Provenance
  • What the engine declares. provenance is required whenever measurements is present:
    • producer (required, 1–64 characters);
    • producer_version (optional);
    • measured_at (optional, RFC 3339);
    • subject (optional): what was measured, for example provider, model and endpoint;
    • method (optional): how it was measured.
    • Free text is bounded (see 5). It is masked exactly like stderr and final_message: the configured API key and every engine.custom.env value become ***REDACTED***.
  • What skill-up records. The engine.name, the transport, the case, the run and, for multi-turn cases, the turn number. The values are marked engine_reported.
    • skill-up does not verify them, consistent with the trust model in docs/design/custom-engine.md.
    • subject is free text for people to read. Like the provider/model identity in #317, it is never a matching or blocking key.
3. Per-step versus aggregate values
  • per_step. Each entry is one series: an ordered list of points along its declared axis.
    • at is a finite number and strictly increasing within the series: one point per step.
    • Raw repeats or per-chunk timings belong in artifacts.files, which already has its own caps.
  • aggregate. Each entry is one value for the whole session (or for one turn; see below). The engine computes it, and description can say how.
    • skill-up never derives aggregates from series, and never sums or averages measurements across turns, repeats or runs.
    • Statistics across runs belong to the comparison slice.
  • Multi-turn cases. Measurements stay attached to the turn whose SessionResult reported them. They are persisted per turn, with no carry-over or merging between turns.
4. Missing and invalid values
  • Missing values. An entry or point carries one of two things:
    • a value, which is a finite JSON number, optionally with "availability": "AVAILABLE" as in the #347 grammar;
    • or "availability": "MISSING" with a reason code matching ^[a-z][a-z0-9_]{0,63}$, and no value.
    • A missing value is never stored or shown as 0, never interpolated, and never dropped silently.
    • A timed-out request at 262k context is information, not noise.
  • Whole-object defects reject the entire measurements object. These are:
    • a wrong or unsupported schema_version;
    • a non-object;
    • duplicate JSON keys at any level;
    • an unknown field at any level;
    • missing or invalid provenance;
    • exceeding a whole-object limit.
  • Entry defects reject only that entry; valid siblings stay usable. These are:
    • an invalid id or unit;
    • a duplicated id: every entry carrying it is rejected, since the value is ambiguous;
    • a non-number or non-finite value;
    • a value together with MISSING, or neither a value nor MISSING;
    • a bad point (non-finite or non-increasing at, or an invalid value): it rejects its whole series;
    • a series over the point limit.
  • Every rejection is recorded as a measurement diagnostic, with the entry id when known and a reason code.
  • Never a verdict. The SessionResult stays valid and the case verdict is unchanged. An informational field must not be able to turn a behavioral evaluation into an ERROR.
5. Validation and size limits
Limit Value On overflow
Serialized measurements object 1 MiB (same as the #347 metrics payload) reject the whole object
Entries (aggregate + per_step) 64 reject the whole object
Points per series 1,000 reject that series
Points per result 10,000 reject the whole object
provenance.method 2 KiB reject the whole object
provenance.subject, producer_version 512 bytes reject the whole object
description per entry 256 bytes reject that entry

Values are never truncated or clamped. A cut series silently changes its meaning, so it is rejected with a diagnostic instead.

v1 has no free-form extra map:

  • every number has a typed home in aggregate or per_step;
  • text goes in description, method or subject;
  • bulky raw data goes in artifacts.files.

This keeps validation and masking complete. If a need for open fields appears, it can come with schema_version: 2 and its own limits.

Persistence and effect
  • Where results are persisted:
    • Validated measurements and their diagnostics go into result.json and report.json.
    • Single-turn cases carry them on the case result. Multi-turn cases carry them on each turn result.
    • Both fields are omitted when the engine sent nothing, so results without measurements are byte-identical to today.
  • What stays untouched: grading.json, benchmark.json, JUnit output, OTel attributes and the JSONL evaluation-event stream. The event stream needs its own schema and consumer review, as noted in #317.
  • No effect on scoring: Measurements never change PASS/FAIL/ERROR/SKIP. They never enter pass rates, benchmark denominators or #347 metric states. There are no thresholds on measurements in this proposal.
  • Built-in agents do not emit measurements in this slice.
  • Older skill-up versions already ignore unknown SessionResult fields, because the custom engine result is decoded without DisallowUnknownFields. Engines can start sending measurements before skill-up reads it.

Slices

  1. Contract and persistence (this issue). Parse, validate, mask and persist the data, with docs in docs/design/custom-engine.md, docs/guide/writing-evals.md and docs/zh/guide/writing-evals.md.
  2. Reports. An "Engine measurements (informational)" section in HTML/Markdown, kept visually apart from verdicts and scores. Aggregates go in a table and series in a table, with charts optional.
  3. History and comparison. Build on #349 / #353, where a measurement's identity is (id, unit) plus the axis (id, unit) for series. Different subject or model identities give an advisory, not a block.
  4. Optional, only if needed. An explicit, evaluation-defined rule that turns a measurement into a score, under the #317 contract.

Acceptance criteria (slice 1)

  • A custom engine on the local and on the http transport can return at least one aggregate and one series; both are persisted with skill-up provenance (engine, case, run, turn).
  • A SessionResult without measurements produces byte-identical result.json and report.json.
  • Verdicts, pass rates, benchmark output and #347 metric states are identical with and without measurements, including when the measurements are invalid.
  • MISSING values keep their reason, and nothing missing becomes 0. Non-number and non-finite values are rejected.
  • Whole-object defects reject the whole object; entry defects reject only their entry while siblings stay; every rejection leaves a diagnostic.
  • Every limit is tested at the boundary and one past it; nothing is truncated.
  • Free-text fields are masked like stderr; ids, units and reason codes outside their patterns are rejected.
  • Multi-turn: measurements stay on the turn that reported them.
  • Docs include the #296 sweep as the worked example.

Open questions

  1. Should slice 1 also add a minimal report table, or stay persistence-only like the first slice of #347?
  2. Are the proposed limits reasonable for real engines? A 300-step agent session with a few series per step fits; anything per-chunk does not, and should go to artifacts.files.
  3. For units, is a validated free-form label with recommended spellings enough, or would you prefer a closed list from the start?

Related

  • #296: the decode-speed decay scenario. A first producer exists: eval_tps_run in dsh-eval-harness, which can be wrapped as a custom engine.
  • #317 / #347: named quality metrics. Separate contract, untouched by this proposal.
  • #349 / #353: history and comparison. Measurements join them in slice 3.
  • Harbor ATIF: the per-step Metrics and FinalMetrics models are the closest reference. They fix a few known fields (tokens, cost) and put anything else in an untyped extra map without units. This proposal keeps the per-step/aggregate split but types every value with a name and a unit.
Lingua principale
Go
Stelle
1.5k
Fork
134
Merge medio
1g 22h
PR unite (30g)
61

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di alibaba/skill-up

Tutte le issue di alibaba/skill-up

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.