Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Proposal: load an FTW with fp16 NVFP4 global scales (upcast with a warning) instead of refusing it after #739

Abierto
  • ¿Algo no está bien?
#769 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
backend, machine-learning

Línea de trabajo

Start with python/freetoken/checkpoint/ftw.py and trace iter_ftw_weights, load_ftw_banks, and load_ftw_banks_to_device, including the bank allocation and read paths. Read the existing NVFP4 dtype checks and the scripts/ftw_hotfix.py and docs/models.md guidance. Add coverage for loading fp16 global scales as fp32, including the warning and inference-mode behavior; done means the load succeeds with the stated values and the docs describe when to reconvert.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Proposal

Since #739, FTWReader refuses every NVFP4 FTW that stores its global scales as fp16, which is every NVFP4 FTW converted before #739. The only fix it offers is a full reconversion from the source checkpoint. Not every user still has that checkpoint. For Qwen3.8-Flash-Next-NVFP4 a reconversion means downloading about 120 GB again and converting it, and until then the server will not start at all.

The fp16 rounding cannot be undone (as #739 says), but it does not have to block loading. Upcasting the stored fp16 values to fp32 at load gives exactly the numerics every pre-#739 build served from the same file. Proposal:

  • FTWReader logs a one-time warning ("this FTW stores NVFP4 global scales as fp16 (converted before #739); serving them upcast to fp32, which keeps the old precision; reconvert with ft checkpoint to get the fp32 globals"), not a ValueError.
  • The dense, host-bank and device-bank read paths read an fp16 global entry into a staging buffer and copy it into the fp32 destination.
  • scripts/ftw_hotfix.py and docs/models.md describe the warning and the reason to reconvert, not a hard failure.

Users get the precision fix by reconverting when they can, and their existing checkpoints keep working until they do.

Minimal reproduction of the refusal (CPU only, PYTHONPATH=python):

import json, os, tempfile
from freetoken.checkpoint.ftw import FTWReader, INDEX_NAME, FORMAT_TAG
d = tempfile.mkdtemp()
with open(os.path.join(d, INDEX_NAME), "w") as f:  # index of an NVFP4 FTW written before #739
    json.dump({"format": FORMAT_TAG, "shards": [], "tensors": [
        {"name": "gate_up_global#L00000", "dtype": "float16", "shape": [256, 1024]}]}, f)
try:
    FTWReader(d)
except ValueError as e:
    print("ValueError:", e)

ValueError: '/tmp/user/1000/tmps9e9_aju' stores NVFP4 global scales as fp16 (e.g. 'gate_up_global#L00000') but FreeToken reads them as fp32; reconvert it from the source checkpoint with `ft checkpoint`
Cause (current main, cc56b6bd)
  • python/freetoken/checkpoint/ftw.py:229-234: FTWReader.__init__ raises on any fp16 entry that _is_nvfp4_global (:105-107) matches.
  • The readers copy raw bytes into banks sized from the index dtype: iter_ftw_weights (:376), and load_ftw_banks _read_row / _read_layer (:614-625), which use reader.read_into(bank.memoryview(), ...). load_ftw_banks_to_device (:707) does the same. So an upcast needs an fp32 destination plus an fp16 staging read for these entries. It cannot just change the dtype the reader reports.
Proposed fix
  • Replace the ValueError with a one-time logger.warning. Record which global entries are fp16.
  • In the three read paths, allocate fp32 for those entries, read the fp16 bytes into a transient buffer, and copy_ (upcast) into the bank. The scheduler builds the engine, and with it the host banks, under torch.inference_mode() (server/launch.py:79-81), and the ThreadPoolExecutor reader threads in load_ftw_banks do not inherit that mode. The in-place copy in those threads must therefore run inside with torch.inference_mode():, or it raises on inference tensors.
  • A test: write a small FTW with fp16 globals, load it under torch.inference_mode() the way the engine does, and check that the warning appears once and that the globals come back as fp32, equal to the fp16 values.
Environment

Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 from an FTW converted before #739, on an RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130; reproduction CPU-only on main cc56b6bd.

Lenguaje dominante
Python
Estrellas
14.2k
Forks
1.4k
Merge medio
21 h 39 min
PR fusionados (30 d)
100

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de FlashML-org/FreeToken

Todos los issues de FlashML-org/FreeToken

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.