feat(engine): optional runtime measurements reported by custom engines
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 35/100
Direzione di ricerca
Start with docs/design/custom-engine.md and locate the custom engine SessionResult decoding and result persistence paths for the local and http transports. The issue defines validation, masking, diagnostics, size limits, and multi-turn requirements, so use its slice 1 acceptance criteria to map the work and identify the relevant tests. Done means valid measurements and diagnostics persist without changing verdicts or outputs, and the three named guides document the #296 sweep.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Problem
A custom engine can report a few fixed numbers about its run in SessionResult: duration_ms, turns, input_tokens and output_tokens. It cannot report any other runtime measurement with a name and a unit, such as decode throughput at given context lengths, time to first token, or effective context length.
Today the only way to carry such numbers is to put a file in artifacts.files. skill-up then archives the file without reading it. Nothing is validated, and nothing reaches result.json, report.json or a later comparison.
The motivating case is #296: long-context decode-speed decay. An instrumented agent or harness, wrapped as a custom engine, measures:
- decode TPS per request at fixed context steps (8k, 16k, 32k, …);
- derived scalars: the baseline speed and the half-speed point n½, the context length where decode speed falls to half of baseline.
As agreed in #296, the measurement and its fitting stay in the external implementation. skill-up only needs to receive, validate, persist and show the results.
The discussion in #317 settled the boundary:
- These are measurements of what happened during execution, not quality scores.
- They stay separate from the
[0,1]named-metric contract of #347. - They do not affect PASS/FAIL.
- They are not aggregated with quality scores by default.
- A measurement can feed a score only through a scoring rule that an evaluation defines explicitly.
This issue proposes that separate contract.
Proposal
Add an optional measurements object to the custom engine SessionResult, for both the local and http transports. The first slice validates and persists it. Showing it in the HTML/Markdown reports and comparing it across runs come later; see Slices.
{
"exit_code": 0,
"final_message": "Decay sweep finished.",
"measurements": {
"schema_version": 1,
"provenance": {
"producer": "dsh-eval-harness",
"producer_version": "e90c255",
"measured_at": "2026-11-17T08:00:00Z",
"subject": "provider=example model=example-model (the deployment as measured, not the architecture)",
"method": "Streaming requests at nominal context steps; a per-request nonce defeats prefix caching; temperature 0, max_tokens 256; median of 3 repeats per step."
},
"aggregate": [
{"id": "baseline_decode_tps", "unit": "tok/s", "value": 200.0,
"description": "1/a of the least-squares fit tps(n) = 1/(a + b*n)"},
{"id": "half_speed_context", "unit": "token", "value": 67000,
"description": "context length where the fitted speed is half of baseline"}
],
"per_step": [
{
"id": "decode_tps",
"unit": "tok/s",
"axis": {"id": "context_tokens", "unit": "token"},
"points": [
{"at": 8192, "value": 178.2},
{"at": 16384, "value": 160.7},
{"at": 32768, "value": 134.3},
{"at": 65536, "value": 101.1},
{"at": 131072, "value": 67.7},
{"at": 262144, "availability": "MISSING", "reason": "request_timed_out"}
]
}
]
}
}
The five points raised in #317 are covered below.
1. Names and units
- Names. Every entry has an
idmatching the judge-id pattern,^[a-z][a-z0-9_-]{0,63}$.- An id is unique within one
measurementsobject, acrossaggregateandper_steptogether. - Measurement ids live in their own namespace. They never collide with #347 metric ids, and reports label the two kinds separately.
- The term "metric" stays reserved for #347 scores. Harbor uses "metrics" more broadly: rewards aggregated across trials, plus the operational numbers in ATIF.
- An id is unique within one
- Units.
unitis required. It is an opaque, case-sensitive label of 1–32 characters from[A-Za-z0-9/%._^*-], compared exactly.- Recommended spellings:
s,ms,token,tok/s,byte,%,usd, and1for dimensionless values. - skill-up never converts units. The same
idwith a different unit is a different measurement.
- Recommended spellings:
- Axis. A
per_stepentry declares what a step is measured along, with its ownidandunit:{"id": "context_tokens", "unit": "token"}for a context sweep;{"id": "step", "unit": "1"}for agent steps;{"id": "elapsed", "unit": "s"}for a time series.
2. Provenance
- What the engine declares.
provenanceis required whenevermeasurementsis present:producer(required, 1–64 characters);producer_version(optional);measured_at(optional, RFC 3339);subject(optional): what was measured, for example provider, model and endpoint;method(optional): how it was measured.- Free text is bounded (see 5). It is masked exactly like
stderrandfinal_message: the configured API key and everyengine.custom.envvalue become***REDACTED***.
- What skill-up records. The
engine.name, the transport, the case, the run and, for multi-turn cases, the turn number. The values are markedengine_reported.- skill-up does not verify them, consistent with the trust model in
docs/design/custom-engine.md. subjectis free text for people to read. Like the provider/model identity in #317, it is never a matching or blocking key.
- skill-up does not verify them, consistent with the trust model in
3. Per-step versus aggregate values
per_step. Each entry is one series: an ordered list of points along its declared axis.atis a finite number and strictly increasing within the series: one point per step.- Raw repeats or per-chunk timings belong in
artifacts.files, which already has its own caps.
aggregate. Each entry is one value for the whole session (or for one turn; see below). The engine computes it, anddescriptioncan say how.- skill-up never derives aggregates from series, and never sums or averages measurements across turns, repeats or runs.
- Statistics across runs belong to the comparison slice.
- Multi-turn cases. Measurements stay attached to the turn whose
SessionResultreported them. They are persisted per turn, with no carry-over or merging between turns.
4. Missing and invalid values
- Missing values. An entry or point carries one of two things:
- a
value, which is a finite JSON number, optionally with"availability": "AVAILABLE"as in the #347 grammar; - or
"availability": "MISSING"with areasoncode matching^[a-z][a-z0-9_]{0,63}$, and novalue. - A missing value is never stored or shown as
0, never interpolated, and never dropped silently. - A timed-out request at 262k context is information, not noise.
- a
- Whole-object defects reject the entire
measurementsobject. These are:- a wrong or unsupported
schema_version; - a non-object;
- duplicate JSON keys at any level;
- an unknown field at any level;
- missing or invalid
provenance; - exceeding a whole-object limit.
- a wrong or unsupported
- Entry defects reject only that entry; valid siblings stay usable. These are:
- an invalid
idorunit; - a duplicated
id: every entry carrying it is rejected, since the value is ambiguous; - a non-number or non-finite value;
- a
valuetogether withMISSING, or neither avaluenorMISSING; - a bad point (non-finite or non-increasing
at, or an invalid value): it rejects its whole series; - a series over the point limit.
- an invalid
- Every rejection is recorded as a measurement diagnostic, with the entry id when known and a reason code.
- Never a verdict. The
SessionResultstays valid and the case verdict is unchanged. An informational field must not be able to turn a behavioral evaluation into anERROR.
5. Validation and size limits
| Limit | Value | On overflow |
|---|---|---|
Serialized measurements object |
1 MiB (same as the #347 metrics payload) | reject the whole object |
Entries (aggregate + per_step) |
64 | reject the whole object |
| Points per series | 1,000 | reject that series |
| Points per result | 10,000 | reject the whole object |
provenance.method |
2 KiB | reject the whole object |
provenance.subject, producer_version |
512 bytes | reject the whole object |
description per entry |
256 bytes | reject that entry |
Values are never truncated or clamped. A cut series silently changes its meaning, so it is rejected with a diagnostic instead.
v1 has no free-form extra map:
- every number has a typed home in
aggregateorper_step; - text goes in
description,methodorsubject; - bulky raw data goes in
artifacts.files.
This keeps validation and masking complete. If a need for open fields appears, it can come with schema_version: 2 and its own limits.
Persistence and effect
- Where results are persisted:
- Validated measurements and their diagnostics go into
result.jsonandreport.json. - Single-turn cases carry them on the case result. Multi-turn cases carry them on each turn result.
- Both fields are omitted when the engine sent nothing, so results without measurements are byte-identical to today.
- Validated measurements and their diagnostics go into
- What stays untouched:
grading.json,benchmark.json, JUnit output, OTel attributes and the JSONL evaluation-event stream. The event stream needs its own schema and consumer review, as noted in #317. - No effect on scoring: Measurements never change PASS/FAIL/ERROR/SKIP. They never enter pass rates, benchmark denominators or #347 metric states. There are no thresholds on measurements in this proposal.
- Built-in agents do not emit measurements in this slice.
- Older skill-up versions already ignore unknown
SessionResultfields, because the custom engine result is decoded withoutDisallowUnknownFields. Engines can start sendingmeasurementsbefore skill-up reads it.
Slices
- Contract and persistence (this issue). Parse, validate, mask and persist the data, with docs in
docs/design/custom-engine.md,docs/guide/writing-evals.mdanddocs/zh/guide/writing-evals.md. - Reports. An "Engine measurements (informational)" section in HTML/Markdown, kept visually apart from verdicts and scores. Aggregates go in a table and series in a table, with charts optional.
- History and comparison. Build on #349 / #353, where a measurement's identity is (
id,unit) plus the axis (id,unit) for series. Differentsubjector model identities give an advisory, not a block. - Optional, only if needed. An explicit, evaluation-defined rule that turns a measurement into a score, under the #317 contract.
Acceptance criteria (slice 1)
- A custom engine on the
localand on thehttptransport can return at least one aggregate and one series; both are persisted with skill-up provenance (engine, case, run, turn). - A
SessionResultwithoutmeasurementsproduces byte-identicalresult.jsonandreport.json. - Verdicts, pass rates, benchmark output and #347 metric states are identical with and without measurements, including when the measurements are invalid.
-
MISSINGvalues keep their reason, and nothing missing becomes0. Non-number and non-finite values are rejected. - Whole-object defects reject the whole object; entry defects reject only their entry while siblings stay; every rejection leaves a diagnostic.
- Every limit is tested at the boundary and one past it; nothing is truncated.
- Free-text fields are masked like
stderr; ids, units and reason codes outside their patterns are rejected. - Multi-turn: measurements stay on the turn that reported them.
- Docs include the #296 sweep as the worked example.
Open questions
- Should slice 1 also add a minimal report table, or stay persistence-only like the first slice of #347?
- Are the proposed limits reasonable for real engines? A 300-step agent session with a few series per step fits; anything per-chunk does not, and should go to
artifacts.files. - For units, is a validated free-form label with recommended spellings enough, or would you prefer a closed list from the start?
Related
- #296: the decode-speed decay scenario. A first producer exists:
eval_tps_runin dsh-eval-harness, which can be wrapped as a custom engine. - #317 / #347: named quality metrics. Separate contract, untouched by this proposal.
- #349 / #353: history and comparison. Measurements join them in slice 3.
- Harbor ATIF: the per-step
MetricsandFinalMetricsmodels are the closest reference. They fix a few known fields (tokens, cost) and put anything else in an untypedextramap without units. This proposal keeps the per-step/aggregate split but types every value with a name and a unit.
- Lingua principale
- Go
- Stelle
- 1.5k
- Fork
- 134
- Merge medio
- 1g 22h
- PR unite (30g)
- 61
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di alibaba/skill-up
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remoteForse già presa @bababoom721 l’ha presa 1 giorno fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
alibaba/skill-up#408 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 15/100
alibaba/skill-up#407 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
alibaba/skill-up#406 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
alibaba/skill-up#405 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di alibaba/skill-up
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
fluxcd/notification-controller#1441 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
OneBusAway/maglev#2145 ·
I maintainer di solito rispondono entro 2 giorni
-
priority/medium status/ready type/chore
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
discobox-ai/discobox#1357 ·
I maintainer di solito rispondono entro 1 giorno
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itForse già presa @fzn0x l’ha presa oggi. Apertabug priority: low
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
edit_file diff rendering panics on invalid UTF-8 textForse già presa @TDnorthgarden l’ha presa oggi. Apertaarea/tui
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
docker/docker-agent#4955 ·
I maintainer di solito rispondono entro 1 giorno