clean/runner/pipeline.py read_wgcm reads every non-float32 tensor as uint8 (int32, float16, uint16, uint32 fail)

未關閉 適合新手
#241 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
2/5
預估耗時
1-3 小時
新手友好度
88/100
Issue 類型
缺陷
描述清晰度
描述清楚
活躍度
活躍
技術堆疊
numpy, python

研究方向

從 clean/runner/pipeline.py 中的 read_wgcm 開始,閱讀 clean/converter/OUTPUT-FORMAT.md 中的張量表,了解 dtype 代碼和記錄佈局。執行該 issue 的 repro 腳本以確認 reshape 失敗,然後將 dtype 分支替換為明確的 code-to-dtype 映射以及位元組數檢查。新增一個單元測試:寫入小容器,每個 dtype 代碼對應一個張量,並斷言解碼後的陣列,以及未知代碼會拋出例外。當 repro 回傳 int32 值且測試涵蓋代碼 0-6 時完成。

由索引模型根據 Issue 內容生成。

描述

bug

Symptom

read_wgcm in clean/runner/pipeline.py maps the tensor dtype code to a numpy type with
dt = np.uint8 if dtype else "<f4": code 0 is read as float32 and every other code as uint8. The container format
(clean/converter/OUTPUT-FORMAT.md, tensor table) has more codes: 0 float32, 1 uint8, 2 int32, and with container
version 2 also 3 float16, 4 uint16, 5 uint32, 6 e4m3.

A preset-K model file has the group net with the int32 tensor window_offsets (shape 8 x 2, 64 bytes). Reading it
fails:

ValueError: cannot reshape array of size 64 into shape (8,2)

so read_wgcm cannot load any Model K model file (and read_model_tensors, which calls it when there is no raw/
folder, fails the same way). float16 / uint16 / uint32 tensors of a version-2 container would also be read with the
wrong element size; e4m3 (one byte) happens to work.

Minimal repro

import struct, numpy as np
from clean.runner.pipeline import read_wgcm

# a one-group container with one int32 tensor [2, 2] (layout per clean/converter/OUTPUT-FORMAT.md; header fields that
# read_wgcm does not use are zero)
name, tname = b"net", b"t"
index = struct.pack("<H", len(name)) + name + bytes(32) + struct.pack("<I", 1)
index += struct.pack("<H", len(tname)) + tname + bytes([2, 2]) + struct.pack("<II", 2, 2)
data_off = 88 + len(index) + 16
index += struct.pack("<QQ", data_off, 16)
header = b"WGCMODEL" + struct.pack("<II", 1, 1) + bytes(64) + struct.pack("<Q", len(index))
open("t.wgcm", "wb").write(header + index + np.arange(4, dtype="<i4").tobytes())
print(read_wgcm("t.wgcm"))

Expected: {"net": {"t": array([[0, 1], [2, 3]], dtype=int32)}}. Actual: ValueError: cannot reshape array of size 16 into shape (2,2).

(Checked on main: the repro prints exactly that error.)

Expected vs actual

  • Expected: each tensor read with the element type of its dtype code; a code the reader does not know raises a clear
    error naming the code.
  • Actual: every nonzero code is read as uint8, so multi-byte types fail to reshape (or, for a size that happens to
    fit, would silently give wrong values).

Proposed fix

Map the codes explicitly and check the byte count:

DTYPES = {0: "<f4", 1: np.uint8, 2: "<i4", 3: "<f2", 4: "<u2", 5: "<u4", 6: np.uint8}
...
if dtype not in DTYPES:
    raise ValueError(f"{path}: tensor {stage}/{name}: unknown dtype code {dtype}")
dt = np.dtype(DTYPES[dtype])
count = int(np.prod(shape)) if shape else 1
if size != count * dt.itemsize:
    raise ValueError(f"{path}: tensor {stage}/{name}: {size} bytes for {count} values of {dt}")
a = np.frombuffer(data[off:off + size], dtype=dt).reshape(shape)
out[stage][name] = a.astype(np.float32) if dtype in (0, 3) else a

and a unit test with one tensor of each dtype code (a container written by the test, no model data). The other
readers (clean/k/tests/test_k_runtime_tables.py parse_wgcm, clean/rr_f/runtime/tools/pack_model.py wgcm_logical,
the converter tests' read_container) may want the same check.

主要語言
C++
星號
12
分支
2
平均合併
3 小時 10 分鐘
30 天內合併 PR
171

環境準備

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

phantoslayer21/wronggpu 的其他 Issue

查看 phantoslayer21/wronggpu 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。