Kumo Forecast mixed-precision predictions diverge from upstream beyond its precision budget
维护者通常 1 天内回复
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 25/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
调研方向
Start by locating Kumo Forecast’s preprocessing, positional value generation, encoder and normalization paths, then compare them with the upstream SDK using the four-feature hourly fixture and the M4, NN5 and synthetic stress inputs described in the issue. Reproduce the mixed-precision discrepancy and isolate the interacting dtype and rounding differences; promoting the encoder stream alone is already known to leave counterexamples. Done means the tested cohort meets the stated agreement target, with remaining FP32 differences, budget exceptions, latency and memory costs, and saved-state key changes documented.
由索引模型根据 Issue 内容生成。
描述
Reader Digest
Summary
Kumo Forecast in SDM can return different predictions from the upstream SDK with the same pretrained weights and histories.
On 20,736 predictions, the mixed-precision discrepancy averaged 1.33 times upstream's own mixed-versus-FP32 discrepancy, with occasional sign changes near zero.
FP32 was much closer, but that does not validate mixed precision.
Tenant, Data, and Specs
- Affected: Local Kumo Forecast inference using the existing pretrained checkpoints; observed in both cross-channel variants.
- Data and connector: Public SDK four-feature hourly fixture, M4 and NN5 histories, and labeled synthetic stress inputs; no connector.
- Machine: NVIDIA L40S, Python 3.12.14, PyTorch 2.12.1+cu130, CUDA 13.0, driver 595.91.07, NumPy 1.26.4.
- Deployment: Source-installed Python packages, public inference APIs; no serving container.
Root Cause
- The port changes preprocessing rounding and regenerates positional values on the GPU.
- Mixed precision changes encoder-stream and normalization dtypes relative to upstream.
- Input layout, GELU and attention execution introduce additional rounding differences.
- These effects interact; promoting the encoder stream alone leaves reproducible counterexamples.
Fix Blast Radius
Keep the correction local to Forecast, with optional division precision in the shared standardizer and unchanged defaults for other models.
The candidate achieves exact mixed agreement in the tested cohort, but retains small FP32 differences and pointwise budget exceptions.
Measured latency/memory costs and saved-state key changes require disclosure; numerical agreement is not forecasting accuracy.
No retraining or checkpoint-weight change is required.
Verdict: Contained. Fixing this addresses the root cause.
- 主要语言
- Python
- 星标
- 429
- 派生
- 46
- 平均合并
- 1 天 13 小时
- 30 天内合并 PR
- 212
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
NVIDIA/structured-data-models 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 70/100
NVIDIA/structured-data-models#1248 ·
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 30/100
NVIDIA/structured-data-models#1276 · 1 条评论 ·
维护者通常 1 天内回复
-
Introduce documentation versioning可能已有人在做 @AkilsuryaS 于 1 天前认领。 未关闭documentation enhancement
难度 5/5 一周以上 新手友好度 35/100
NVIDIA/structured-data-models#1246 · 1 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
-
NIM error可能已有人在做 @am-porter 于 9 天前认领。 未关闭question
难度 4/5 1-3 小时 新手友好度 15/100
NVIDIA/structured-data-models#1136 · 3 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
-
Update `Recipe` contract to allow for more generic transformations可能已有人在做 @rusty1s 于 9 天前认领。 未关闭enhancement
难度 5/5 一周以上 新手友好度 35/100
NVIDIA/structured-data-models#1071 · 1 个 reaction · 已指派 2 人 ·
维护者通常 1 天内回复
查看 NVIDIA/structured-data-models 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 79/100
aws-samples/sample-per-model-bedrock#639 ·
维护者通常 1 天内回复
-
bug
难度 1/5 1 小时以内 新手友好度 86/100
awslabs/visual-asset-management-system#937 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 79/100
project-codeflare/codeflare-sdk#1195 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 65/100
pandas-dev/pandas-stubs#2051 ·
维护者通常 1 天内回复