Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Proposal: load an FTW with fp16 NVFP4 global scales (upcast with a warning) instead of refusing it after #739

オープン
  • 何か問題がありますか？
#769 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python, pytorch
領域
backend, machine-learning

調査の方向性

Start with python/freetoken/checkpoint/ftw.py and trace iter_ftw_weights, load_ftw_banks, and load_ftw_banks_to_device, including the bank allocation and read paths. Read the existing NVFP4 dtype checks and the scripts/ftw_hotfix.py and docs/models.md guidance. Add coverage for loading fp16 global scales as fp32, including the warning and inference-mode behavior; done means the load succeeds with the stated values and the docs describe when to reconvert.

索引モデルが issue の本文から書いたものです。

説明

Proposal

Since #739, FTWReader refuses every NVFP4 FTW that stores its global scales as fp16, which is every NVFP4 FTW converted before #739. The only fix it offers is a full reconversion from the source checkpoint. Not every user still has that checkpoint. For Qwen3.8-Flash-Next-NVFP4 a reconversion means downloading about 120 GB again and converting it, and until then the server will not start at all.

The fp16 rounding cannot be undone (as #739 says), but it does not have to block loading. Upcasting the stored fp16 values to fp32 at load gives exactly the numerics every pre-#739 build served from the same file. Proposal:

  • FTWReader logs a one-time warning ("this FTW stores NVFP4 global scales as fp16 (converted before #739); serving them upcast to fp32, which keeps the old precision; reconvert with ft checkpoint to get the fp32 globals"), not a ValueError.
  • The dense, host-bank and device-bank read paths read an fp16 global entry into a staging buffer and copy it into the fp32 destination.
  • scripts/ftw_hotfix.py and docs/models.md describe the warning and the reason to reconvert, not a hard failure.

Users get the precision fix by reconverting when they can, and their existing checkpoints keep working until they do.

Minimal reproduction of the refusal (CPU only, PYTHONPATH=python):

import json, os, tempfile
from freetoken.checkpoint.ftw import FTWReader, INDEX_NAME, FORMAT_TAG
d = tempfile.mkdtemp()
with open(os.path.join(d, INDEX_NAME), "w") as f:  # index of an NVFP4 FTW written before #739
    json.dump({"format": FORMAT_TAG, "shards": [], "tensors": [
        {"name": "gate_up_global#L00000", "dtype": "float16", "shape": [256, 1024]}]}, f)
try:
    FTWReader(d)
except ValueError as e:
    print("ValueError:", e)

ValueError: '/tmp/user/1000/tmps9e9_aju' stores NVFP4 global scales as fp16 (e.g. 'gate_up_global#L00000') but FreeToken reads them as fp32; reconvert it from the source checkpoint with `ft checkpoint`
Cause (current main, cc56b6bd)
  • python/freetoken/checkpoint/ftw.py:229-234: FTWReader.__init__ raises on any fp16 entry that _is_nvfp4_global (:105-107) matches.
  • The readers copy raw bytes into banks sized from the index dtype: iter_ftw_weights (:376), and load_ftw_banks _read_row / _read_layer (:614-625), which use reader.read_into(bank.memoryview(), ...). load_ftw_banks_to_device (:707) does the same. So an upcast needs an fp32 destination plus an fp16 staging read for these entries. It cannot just change the dtype the reader reports.
Proposed fix
  • Replace the ValueError with a one-time logger.warning. Record which global entries are fp16.
  • In the three read paths, allocate fp32 for those entries, read the fp16 bytes into a transient buffer, and copy_ (upcast) into the bank. The scheduler builds the engine, and with it the host banks, under torch.inference_mode() (server/launch.py:79-81), and the ThreadPoolExecutor reader threads in load_ftw_banks do not inherit that mode. The in-place copy in those threads must therefore run inside with torch.inference_mode():, or it raises on inference tensors.
  • A test: write a small FTW with fp16 globals, load it under torch.inference_mode() the way the engine does, and check that the warning appears once and that the globals come back as fp32, equal to the fp16 values.
Environment

Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 from an FTW converted before #739, on an RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130; reproduction CPU-only on main cc56b6bd.

主要言語
Python
スター
14.2k
フォーク
1.4k
平均マージ
21時間 39分
マージ済み PR（30日）
100

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

FlashML-org/FreeToken のほかの issue

FlashML-org/FreeToken の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。