Real INT8 W8A8 matmul on tensor cores
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 53/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- machine-learning, performance
Research direction
Read src/bitsmith/nn/qlinear.py around set_backend and forward to understand the existing reference path, then inspect tests/gpu/test_cuda.py for its comparison tests. Implement the INT8 activation path with the stated shape-based fallback, and compare its output and W8A8 prefill speed against the reference and fp16 on Ampere; done means it meets the tested error bound and is faster.
Written by the indexing model from the issue text.
Description
w8a8 and w8a8+smoothquant currently simulate activation quantization (fake quant) and
then run a float matmul, so they are slower than plain INT8 weights.
Task: add a backend for int8 layers with act_bits=8 that quantizes activations per
token to int8 and calls torch._int_mm (int8 x int8 -> int32 on tensor cores), then
applies the activation and weight scales. Note torch._int_mm has shape requirements
(rows above 16, dimensions multiples of 8), so fall back to the reference path otherwise.
Files
src/bitsmith/nn/qlinear.py:set_backendandforwardtests/gpu/test_cuda.py: compare with the reference path
Done when W8A8 prefill is faster than fp16 on an Ampere GPU and matches the reference
within a tested error bound.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from yashwanthdevavarapu/bitsmith
-
area: benchmarks good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
area: benchmarks good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
area: cli area: formats good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
-
area: cli good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
area: benchmarks good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
All issues in yashwanthdevavarapu/bitsmith
Similar issues
-
fuzz fuzz-unconfirmed
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Switch to ruffOpen
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Qiskit/qiskit-aer#2533 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
etesync/etesync-dav#376 ·