Kumo Forecast mixed-precision predictions diverge from upstream beyond its precision budget

Cerrado
  • ¿Algo no está bien?
#1,239 5 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@devanj08 ya está trabajando en esto.

Desde el 22/10/2026.

  • #1240 de @devanj08 — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
numpy, python, pytorch
Área
machine-learning

Línea de trabajo

Start by locating Kumo Forecast’s preprocessing, positional value generation, encoder and normalization paths, then compare them with the upstream SDK using the four-feature hourly fixture and the M4, NN5 and synthetic stress inputs described in the issue. Reproduce the mixed-precision discrepancy and isolate the interacting dtype and rounding differences; promoting the encoder stream alone is already known to leave counterexamples. Done means the tested cohort meets the stated agreement target, with remaining FP32 differences, budget exceptions, latency and memory costs, and saved-state key changes documented.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

Reader Digest

Summary

Kumo Forecast in SDM can return different predictions from the upstream SDK with the same pretrained weights and histories.
On 20,736 predictions, the mixed-precision discrepancy averaged 1.33 times upstream's own mixed-versus-FP32 discrepancy, with occasional sign changes near zero.
FP32 was much closer, but that does not validate mixed precision.

Tenant, Data, and Specs
  • Affected: Local Kumo Forecast inference using the existing pretrained checkpoints; observed in both cross-channel variants.
  • Data and connector: Public SDK four-feature hourly fixture, M4 and NN5 histories, and labeled synthetic stress inputs; no connector.
  • Machine: NVIDIA L40S, Python 3.12.14, PyTorch 2.12.1+cu130, CUDA 13.0, driver 595.91.07, NumPy 1.26.4.
  • Deployment: Source-installed Python packages, public inference APIs; no serving container.
Root Cause
  1. The port changes preprocessing rounding and regenerates positional values on the GPU.
  2. Mixed precision changes encoder-stream and normalization dtypes relative to upstream.
  3. Input layout, GELU and attention execution introduce additional rounding differences.
  4. These effects interact; promoting the encoder stream alone leaves reproducible counterexamples.
Fix Blast Radius

Keep the correction local to Forecast, with optional division precision in the shared standardizer and unchanged defaults for other models.
The candidate achieves exact mixed agreement in the tested cohort, but retains small FP32 differences and pointwise budget exceptions.
Measured latency/memory costs and saved-state key changes require disclosure; numerical agreement is not forecasting accuracy.
No retraining or checkpoint-weight change is required.
Verdict: Contained. Fixing this addresses the root cause.

Lenguaje dominante
Python
Estrellas
429
Forks
46
Merge medio
1 d 6 h
PR fusionados (30 d)
227

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/structured-data-models

Todos los issues de NVIDIA/structured-data-models

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.