[Ingest] guard against coverage regressions when a narrow sweep supersedes a published curve / 防止窄矩阵 sweep 替换已发布曲线导致覆盖面退化

Abierto
  • ¿Algo no está bien?
#1,185 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
50/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
typescript
Área
backend, data, databases

Línea de trabajo

Start by tracing the ingest pre-commit path and the published-curve data in the database; packages/db/src/etl/run-overrides.ts shows the current post-publication remedy. Define the coverage comparison for arms, per-arm concurrency spans, and point counts, then validate it against the H200 example so narrower incoming coverage is surfaced before publication.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

A sweep run that validates a code change is usually deliberately narrow. When such a run is ingested
it becomes the latest date for its hardware, and the dashboard renders the latest date — so a narrow
validation sweep can silently replace a denser published curve. Nothing in the pipeline compares the
incoming coverage against what is already published, so every automated signal stays green.

This happened three times on 2026-09-19, from three PRs merged the same morning by sweep-reuse:

PR run hardware
SemiAnalysisAI/InferenceX#3206 35165441471 GB300 aggregated
SemiAnalysisAI/InferenceX#3047 34819564154 GB300 disaggregated
SemiAnalysisAI/InferenceX#3053 34819578354 H200

It was first reported externally — someone at a lab asked why the GB300 Kimi-K3 numbers looked
wrong.

Worked example (fully measured)

H200 Kimi-K3, before and after the 34819578354 ingest:

date arms points
2026-09-13 TP16/EP32 offload=off c1–c12; TP8/EP32 offload=off c1–c16; TP8/EP32 offload=on c8–c32 35
2026-09-19 TP8/EP32 offload=off c1–c16 12

The published curve lost two arms and the pareto front was truncated at c16 instead of c32. The
surviving arm's values match the 2026-09-13 arm it duplicates to within ±6% at every shared
concurrency, so this was purely a coverage regression, not a measurement problem. Purged in #1183.

Why nothing caught it

  1. The check exists only as prose. The InferenceX PR policy says to compare a candidate's points
    and interactivity span against the prior curve and establish replace-versus-append intent before
    merging. There is no gate, so skipping it is silent.
  2. Every automated signal was green. The sweeps concluded success, the reuse gate passed, merge
    checks passed. The pipeline only asks whether the declared matrix passed — never whether the
    incoming coverage is narrower than what it supersedes.
  3. Narrow was correct for those PRs. All three were recipe/code changes enabling measured power.
    A small matrix is the right way to validate that cheaply. The problem is that ingest treats a
    validation sweep identically to a data-refresh sweep.
  4. The consequence is invisible from the PR. Nothing in the InferenceX PR shows what the
    published curve currently looks like, so catching this requires manually opening the dashboard
    before and after — for each merge.
  5. Latest-date-per-hardware rendering. This display rule is what converts "a small new date" into
    "the denser curve disappears". Nobody is thinking about it at merge time.

Proposed check

Before ingest commits a run, compare its coverage against the currently published set for the same
(model, hardware):

  • arms — the set of distinct configurations (tp/ep/dp-attention/offload/disagg topology)
  • concurrency span — min and max concurrency per arm
  • point count per arm

If the incoming set covers fewer arms, or narrows an arm's concurrency span, treat it as a
coverage regression. Surface it rather than silently publishing: fail the ingest with a clear
diff, or ingest but flag the run and require an explicit replaces-published-curve acknowledgement
on the PR.

A point-count floor alone is not sufficient. The proposal circulating in Slack is "at least 5 points
on the pareto"; the H200 case had 12 points and still hid a 35-point set. The quantity that matters
is coverage relative to what is already published, not an absolute count.

Notes

  • The comparison should be computable entirely from data already in the DB plus the incoming run, so
    it can run pre-commit inside ingest rather than as a post-hoc dashboard check.
  • PURGED_RUNS in packages/db/src/etl/run-overrides.ts is the current remedy and it works, but it
    is applied after the regression has been published and noticed by a reader.

中文说明

用于验证代码改动的 sweep 通常是有意收窄的。这类运行一旦导入，就会成为对应硬件的最新日期，而
仪表板按最新日期渲染 —— 于是一个窄的验证性 sweep 会无声地替换掉更密的已发布曲线。流水线中没有
任何环节会把即将导入的覆盖面与已发布的曲线作比较，因此所有自动信号都是绿的。

2026-09-19 当天发生了三次，来自同一个上午通过 sweep-reuse 合并的三个 PR（见上表）。问题最先由
外部反馈发现 —— 有实验室的人询问 GB300 的 Kimi-K3 数据为何异常。

实测案例： H200 Kimi-K3 在 34819578354 导入前后，已发布曲线从三条支路共 35 个点缩减为一条
支路 12 个点，pareto 前沿从 c32 被截断到 c16。存活支路与它重复的 2026-09-13 支路在每个共享并发
点上数值差异均在 ±6% 以内，因此这纯粹是覆盖面退化，而非测量问题。已在 #1183 中清除。

为何没被发现： 该检查只存在于策略文档的散文里，没有对应闸门；所有自动信号均为绿色，流水线
只检查"声明的矩阵是否跑通"，从不比较覆盖面；窄矩阵对那三个 PR 而言是正确且经济的做法，问题在于
导入环节把验证性 sweep 与数据刷新 sweep 一视同仁；后果在 PR 页面上完全不可见，发现它需要人工在
每次合并前后打开仪表板对比；而"每个硬件取最新日期"的渲染规则，正是把"多了一个小日期"变成"更密
的曲线消失"的那一环。

建议的检查： 在导入落库前，将该运行的覆盖面与同一 (model, hardware) 当前已发布的集合作
比较 —— 支路集合（tp/ep/dp-attention/offload/分离拓扑）、每条支路的并发跨度、以及每条支路的点数。
若支路变少或某条支路的并发跨度收窄，即判定为覆盖面退化，应当显式暴露而非静默发布：或让导入
失败并给出清晰的差异，或允许导入但标记该运行，并要求 PR 上给出明确的"替换已发布曲线"确认。

仅设点数下限并不足够。Slack 中流传的方案是"pareto 上至少 5 个点"，而 H200 那条有 12 个点，照样
遮蔽了 35 个点的集合。真正重要的量是相对于已发布内容的覆盖面，而不是绝对点数。

Lenguaje dominante
TypeScript
Estrellas
49
Forks
22
Merge medio
7 h 50 min
PR fusionados (30 d)
243

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SemiAnalysisAI/InferenceX-app

Todos los issues de SemiAnalysisAI/InferenceX-app

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.