fbgemm-xpu: fuse INT4/INT8 quantized lookup across tables into fewer kernel launches
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- cpp, pytorch
- Domain
- backend, performance
Research direction
Read embedding_forward_quantized.cpp and embedding_forward_quantized_kernel_template.h to trace the per-table INT4 and INT8 launches, then review the upstream CUDA launcher for its table dimension. Run test_int_nbit_lookup.py to check CPU/XPU parity and single-table behavior. Done means launches stop scaling with table count, required mixed-type coverage is added, benchmarks cover 1–128 tables, and the header comment and package docs are updated.
Written by the indexing model from the issue text.
Description
The XPU quantized lookup (int_nbit_split_embedding_codegen_lookup_function) submits one SYCL kernel per embedding table, so launch overhead grows linearly with table count. Process several tables per launch instead.
Background
- Added in #132 (c7d216e).
- Raised in this review thread. Realistic DLRM/TorchRec models have dozens to hundreds of tables.
- The author deferred it to a follow-up because fusion needs a kernel redesign. The reviewer agreed.
- Documented in 339a38c.
Current behavior
embedding_forward_quantized.cpploops over tables and submits oneLookupInt4NobagKernelorLookupInt8NobagKernelfunctor per table. These functors are defined inembedding_forward_quantized_kernel_template.h.- The upstream CUDA launcher puts the table dimension in its grid, so one launch covers all tables.
- Tests are in
test_int_nbit_lookup.py.
Acceptance criteria
- Launches no longer scale with table count. Use one per weight type, or one shared launch.
- CPU/XPU parity for INT4 and INT8.
- Mixed INT4/INT8 tables are tested, if the design supports them.
- No single-table regression.
- Benchmark results for 1–128 tables, using the same BMG methodology as #132.
- The header comment and package docs are updated.
- Dominant language
- C++
- Stars
- 5
- Forks
- 9
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 11
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/torchlib-xpu
-
Difficulty 4/5 3-5 days Newbie friendliness 22/100
intel/torchlib-xpu#155 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 22/100
intel/torchlib-xpu#154 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
intel/torchlib-xpu#153 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
intel/torchlib-xpu#152 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
intel/torchlib-xpu#151 ·
Maintainers usually reply within 3 days
All issues in intel/torchlib-xpu
Similar issues
-
area:decoder area:libnvmpi bug P3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
gjrtimmer/jetson-nvmpi#164 ·
-
LSan reports obj_gen leak on post-parse usage() exits (aarch64)Possibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
redis/memtier_benchmark#850 ·
Maintainers usually reply within 1 day
-
bug-unconfirmed
Difficulty 1/5 Under an hour Newbie friendliness 91/100
Maintainers usually reply within 1 day
-
Status: Needs confirmation Status: Needs triage
Difficulty 1/5 1-3 hours Newbie friendliness 77/100
FreeCAD/FreeCAD#33863 · 1 comment ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pawelsalawa/letos#5839 ·