Misc. bug: MUL_MAT: wrong results, exceeds tolerance; gfx1200 HIP/ROCm; f16 + deterministic.
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 46/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- cpp
- Domain
- machine-learning, testing-qa
Research direction
Start by running the reported test-backend-ops command with the given HIP build configuration on gfx1200 to confirm the failing MUL_MAT case. Then trace the HIP backend path for f16-by-f32 matrix multiplication and inspect its precision and determinism behavior. Done means the reported case passes within the stated tolerance; the issue does not identify a root cause or relevant source file.
Written by the indexing model from the issue text.
Description
Name and Version
0.7.0-dev, ggml commit a37cc46
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
Test code, Other (Please specify in the next section)
Command line
./build-hip/bin/test-backend-ops test -j 1 -o MUL_MAT -p 'n=2105344'
Problem description & steps to reproduce
In MUL_MAT on HIP/ROCm gfx1200, I got wrong results, no crash. This happened on ROCm 7.1.1, Ubuntu 26.04, and RX 9060 XT. ERR = 0.004255280 > 0.000500000 , 8.5x tolerance. Case: MUL_MAT(type_a=f16,type_b=f32,m=16,n=2105344,k=32,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0). Deterministic: 0.004253291 on October 21st, 0.004255280 on October 23rd, same to three significant figures across runs with different random inputs. So it's a systematic precision problem, not a flaky race. Not claimed: No root cause. Repro reports 0/1 tests passed. Build Command:
cmake -S . -B build-hip -DGGML_HIP=ON -DGPU_TARGETS=gfx1200
-DLLAMA_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-hip --target test-backend-ops -j 12
First Bad Commit
Don’t know, present at a37cc46, haven’t bisected.
Relevant log output
[MUL_MAT] ERR = 0.004255280 > 0.000500000 MUL_MAT(type_a=f16,type_b=f32,m=16,n=2105344,k=32,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0): FAIL
- Dominant language
- C++
- Stars
- 132k
- Forks
- 24.6k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 501
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ggml-org/llama.cpp
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
bug-unconfirmed
Difficulty 1/5 Under an hour Newbie friendliness 91/100
Maintainers usually reply within 1 day
-
bug-unconfirmed
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Maintainers usually reply within 1 day
-
Misc. bug: Metal aborts on MUL_MAT with BF16 src1 at K >= 64, batch > 8 (no BF16 mul_mm kernel)Possibly taken @rashid5925 claimed this 1 day ago. Openbug-unconfirmed
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
ggml-org/llama.cpp#30973 · 4 comments ·
Maintainers usually reply within 1 day
-
Windows bundled llama-server.exe is missing `longPathAware`: GGUF paths > 260 chars fail with "No such file or directory" even when the file exists and LongPathsEnabled=1Possibly taken @origin-labs-ai claimed this 1 day ago. Openbug-unconfirmed
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
All issues in ggml-org/llama.cpp
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
bug code-review testing tooling
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
aristocratos/btop#1945 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
boykopovar/AnyPS5#11291 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
k2-fsa/sherpa-onnx#4100 ·
Maintainers usually reply within 2 days