clean/runner/pipeline.py read_wgcm reads every non-float32 tensor as uint8 (int32, float16, uint16, uint32 fail)
維護者通常 1 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 2/5
- 預估耗時
- 1-3 小時
- 新手友好度
- 88/100
- Issue 類型
- 缺陷
- 描述清晰度
- 描述清楚
- 活躍度
- 活躍
- 技術堆疊
- numpy, python
- 領域
- data, machine-learning
研究方向
從 clean/runner/pipeline.py 中的 read_wgcm 開始,閱讀 clean/converter/OUTPUT-FORMAT.md 中的張量表,了解 dtype 代碼和記錄佈局。執行該 issue 的 repro 腳本以確認 reshape 失敗,然後將 dtype 分支替換為明確的 code-to-dtype 映射以及位元組數檢查。新增一個單元測試:寫入小容器,每個 dtype 代碼對應一個張量,並斷言解碼後的陣列,以及未知代碼會拋出例外。當 repro 回傳 int32 值且測試涵蓋代碼 0-6 時完成。
由索引模型根據 Issue 內容生成。
描述
Symptom
read_wgcm in clean/runner/pipeline.py maps the tensor dtype code to a numpy type with
dt = np.uint8 if dtype else "<f4": code 0 is read as float32 and every other code as uint8. The container format
(clean/converter/OUTPUT-FORMAT.md, tensor table) has more codes: 0 float32, 1 uint8, 2 int32, and with container
version 2 also 3 float16, 4 uint16, 5 uint32, 6 e4m3.
A preset-K model file has the group net with the int32 tensor window_offsets (shape 8 x 2, 64 bytes). Reading it
fails:
ValueError: cannot reshape array of size 64 into shape (8,2)
so read_wgcm cannot load any Model K model file (and read_model_tensors, which calls it when there is no raw/
folder, fails the same way). float16 / uint16 / uint32 tensors of a version-2 container would also be read with the
wrong element size; e4m3 (one byte) happens to work.
Minimal repro
import struct, numpy as np
from clean.runner.pipeline import read_wgcm
# a one-group container with one int32 tensor [2, 2] (layout per clean/converter/OUTPUT-FORMAT.md; header fields that
# read_wgcm does not use are zero)
name, tname = b"net", b"t"
index = struct.pack("<H", len(name)) + name + bytes(32) + struct.pack("<I", 1)
index += struct.pack("<H", len(tname)) + tname + bytes([2, 2]) + struct.pack("<II", 2, 2)
data_off = 88 + len(index) + 16
index += struct.pack("<QQ", data_off, 16)
header = b"WGCMODEL" + struct.pack("<II", 1, 1) + bytes(64) + struct.pack("<Q", len(index))
open("t.wgcm", "wb").write(header + index + np.arange(4, dtype="<i4").tobytes())
print(read_wgcm("t.wgcm"))
Expected: {"net": {"t": array([[0, 1], [2, 3]], dtype=int32)}}. Actual: ValueError: cannot reshape array of size 16 into shape (2,2).
(Checked on main: the repro prints exactly that error.)
Expected vs actual
- Expected: each tensor read with the element type of its dtype code; a code the reader does not know raises a clear
error naming the code. - Actual: every nonzero code is read as uint8, so multi-byte types fail to reshape (or, for a size that happens to
fit, would silently give wrong values).
Proposed fix
Map the codes explicitly and check the byte count:
DTYPES = {0: "<f4", 1: np.uint8, 2: "<i4", 3: "<f2", 4: "<u2", 5: "<u4", 6: np.uint8}
...
if dtype not in DTYPES:
raise ValueError(f"{path}: tensor {stage}/{name}: unknown dtype code {dtype}")
dt = np.dtype(DTYPES[dtype])
count = int(np.prod(shape)) if shape else 1
if size != count * dt.itemsize:
raise ValueError(f"{path}: tensor {stage}/{name}: {size} bytes for {count} values of {dt}")
a = np.frombuffer(data[off:off + size], dtype=dt).reshape(shape)
out[stage][name] = a.astype(np.float32) if dtype in (0, 3) else a
and a unit test with one tensor of each dtype code (a container written by the test, no model data). The other
readers (clean/k/tests/test_k_runtime_tables.py parse_wgcm, clean/rr_f/runtime/tools/pack_model.py wgcm_logical,
the converter tests' read_container) may want the same check.
- 主要語言
- C++
- 星號
- 12
- 分支
- 2
- 平均合併
- 3 小時 10 分鐘
- 30 天內合併 PR
- 171
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
phantoslayer21/wronggpu 的其他 Issue
-
難度 2/5 1-3 小時 新手友好度 88/100
phantoslayer21/wronggpu#275 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 82/100
phantoslayer21/wronggpu#274 ·
維護者通常 1 天內回覆
-
bug build
難度 2/5 1-3 小時 新手友好度 74/100
phantoslayer21/wronggpu#234 ·
維護者通常 1 天內回覆
-
bug build
難度 2/5 1-3 小時 新手友好度 80/100
phantoslayer21/wronggpu#233 ·
維護者通常 1 天內回覆
-
area: build-ci build
難度 2/5 1-3 小時 新手友好度 78/100
phantoslayer21/wronggpu#56 ·
維護者通常 1 天內回覆
查看 phantoslayer21/wronggpu 的全部 Issue
相似的 Issue
-
ai_p2
難度 2/5 1-3 小時 新手友好度 80/100
ClickHouse/ClickHouse#132519 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 76/100
juce-framework/JUCE#1810 ·
-
難度 1/5 1 小時以內 新手友好度 82/100
petercorke/robotics-toolbox-python#952 ·
維護者通常 1 天內回覆
-
NatronRenderer: a Python error in the script (-t / project script) doesn't change the exit status未關閉func:engine type:bug
難度 2/5 1-3 小時 新手友好度 72/100
NatronGitHub/Natron#1276 ·
維護者通常 1 天內回覆
-
難度 1/5 1-3 小時 新手友好度 77/100
維護者通常 1 天內回覆