[Issue]: Intel Granite Rapids (GNR) are misdetected with lower UPI InterCpuBw
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- cpp
- Domain
- distributed-systems, performance
Research direction
Start in src/graph/topo.cc at ncclTopoGetInterCpuBw and inspect how Intel CPU model IDs select the inter-socket bandwidth constants. Verify the Granite Rapids model IDs and the proposed 48.0 bandwidth, including whether model 0xAE applies. Done means GNR systems report SYS[48.0] instead of SYS[22.0] and the related all_reduce_perf busbw improves.
Written by the indexing model from the issue text.
Description
How is this issue impacting you?
Lower performance than expected
Share Your Debug Logs
Hello
Current version of ncclTopoGetInterCpuBw function has no support of GNR family of Intel Xeon CPUs
https://github.com/NVIDIA/nccl/blob/v2.30.3-1/src/graph/topo.cc#L73
if (cpu->cpu.arch == NCCL_TOPO_CPU_ARCH_X86 && cpu->cpu.vendor == NCCL_TOPO_CPU_VENDOR_INTEL) {
*bw =
cpu->cpu.model == NCCL_TOPO_CPU_MODEL_INTEL_ERP ? ERP_QPI_BW :
cpu->cpu.model == NCCL_TOPO_CPU_MODEL_INTEL_SRP ? SRP_QPI_BW :
cpu->cpu.model == NCCL_TOPO_CPU_MODEL_INTEL_SKL ? SKL_QPI_BW :
BDW_QPI_BW;
}
I think that familyId == 6 && modelId == 0xAD will detect GNR Xeon chips, and they have UPI speed of 24 GT/s per channel (with multiple UPI links between sockets)
https://www.intel.com/content/www/us/en/products/sku/242668/intel-xeon-6507p-processor-48m-cache-3-50-ghz/specifications.html
I think for NCCL graph this will be GNR_QPI_BW equal to 48.0
Some sources also mention modelId 0xAE as GRANITERAPIDS D, but they are probably single socket only.
Current version may allocate less channels for 2 NUMA GNR machines with multiple PCIe-only GPUs without NVlink. I had 'SYS[22.0]' in NCCL_DEBUG with current code, and 'SYS[48.0]' after fixing, and busbw of all_reduce_perf improved after the fix.
Steps to Reproduce the Issue
No response
NCCL Version
2.30.3
Your platform details
No response
Error Message & Behavior
No response
- Dominant language
- C++
- Stars
- 5.1k
- Forks
- 1.4k
- Avg merge
- 2h 3m
- Merged PRs (30d)
- 2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/nccl
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
-
Difficulty 3/5 1-2 days Newbie friendliness 76/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
AXERA-TECH/ax-llm#77 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
games-on-whales/wolf#509 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
bug-unconfirmed
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
NVIDIA/cuda-samples#453 ·