Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Proposal: load an FTW with fp16 NVFP4 global scales (upcast with a warning) instead of refusing it after #739

Aperta
  • Qualcosa non va?
#769 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, machine-learning

Direzione di ricerca

Start with python/freetoken/checkpoint/ftw.py and trace iter_ftw_weights, load_ftw_banks, and load_ftw_banks_to_device, including the bank allocation and read paths. Read the existing NVFP4 dtype checks and the scripts/ftw_hotfix.py and docs/models.md guidance. Add coverage for loading fp16 global scales as fp32, including the warning and inference-mode behavior; done means the load succeeds with the stated values and the docs describe when to reconvert.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Proposal

Since #739, FTWReader refuses every NVFP4 FTW that stores its global scales as fp16, which is every NVFP4 FTW converted before #739. The only fix it offers is a full reconversion from the source checkpoint. Not every user still has that checkpoint. For Qwen3.8-Flash-Next-NVFP4 a reconversion means downloading about 120 GB again and converting it, and until then the server will not start at all.

The fp16 rounding cannot be undone (as #739 says), but it does not have to block loading. Upcasting the stored fp16 values to fp32 at load gives exactly the numerics every pre-#739 build served from the same file. Proposal:

  • FTWReader logs a one-time warning ("this FTW stores NVFP4 global scales as fp16 (converted before #739); serving them upcast to fp32, which keeps the old precision; reconvert with ft checkpoint to get the fp32 globals"), not a ValueError.
  • The dense, host-bank and device-bank read paths read an fp16 global entry into a staging buffer and copy it into the fp32 destination.
  • scripts/ftw_hotfix.py and docs/models.md describe the warning and the reason to reconvert, not a hard failure.

Users get the precision fix by reconverting when they can, and their existing checkpoints keep working until they do.

Minimal reproduction of the refusal (CPU only, PYTHONPATH=python):

import json, os, tempfile
from freetoken.checkpoint.ftw import FTWReader, INDEX_NAME, FORMAT_TAG
d = tempfile.mkdtemp()
with open(os.path.join(d, INDEX_NAME), "w") as f:  # index of an NVFP4 FTW written before #739
    json.dump({"format": FORMAT_TAG, "shards": [], "tensors": [
        {"name": "gate_up_global#L00000", "dtype": "float16", "shape": [256, 1024]}]}, f)
try:
    FTWReader(d)
except ValueError as e:
    print("ValueError:", e)

ValueError: '/tmp/user/1000/tmps9e9_aju' stores NVFP4 global scales as fp16 (e.g. 'gate_up_global#L00000') but FreeToken reads them as fp32; reconvert it from the source checkpoint with `ft checkpoint`
Cause (current main, cc56b6bd)
  • python/freetoken/checkpoint/ftw.py:229-234: FTWReader.__init__ raises on any fp16 entry that _is_nvfp4_global (:105-107) matches.
  • The readers copy raw bytes into banks sized from the index dtype: iter_ftw_weights (:376), and load_ftw_banks _read_row / _read_layer (:614-625), which use reader.read_into(bank.memoryview(), ...). load_ftw_banks_to_device (:707) does the same. So an upcast needs an fp32 destination plus an fp16 staging read for these entries. It cannot just change the dtype the reader reports.
Proposed fix
  • Replace the ValueError with a one-time logger.warning. Record which global entries are fp16.
  • In the three read paths, allocate fp32 for those entries, read the fp16 bytes into a transient buffer, and copy_ (upcast) into the bank. The scheduler builds the engine, and with it the host banks, under torch.inference_mode() (server/launch.py:79-81), and the ThreadPoolExecutor reader threads in load_ftw_banks do not inherit that mode. The in-place copy in those threads must therefore run inside with torch.inference_mode():, or it raises on inference tensors.
  • A test: write a small FTW with fp16 globals, load it under torch.inference_mode() the way the engine does, and check that the warning appears once and that the globals come back as fp32, equal to the fp16 values.
Environment

Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 from an FTW converted before #739, on an RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130; reproduction CPU-only on main cc56b6bd.

Lingua principale
Python
Stelle
14.2k
Fork
1.4k
Merge medio
21h 17m
PR unite (30g)
97

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FlashML-org/FreeToken

Tutte le issue di FlashML-org/FreeToken

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.