Got NaN for communication time, looking for ncclKernel in the trace
#247 建立於 2025年5月8日
倉庫指標
- 星標
- (535 顆星)
- PR 合併指標
- (30 天內沒有已合併 PR)
描述
What is your question?
I'm using HTA to analyze my tracem which is got from the pytorch profiler. However I got NaN for the communication time, and I dive into the code and I found in the communication_analysis.py, the communication kernel in my trace is 0 (which is not the case since I'm doing multi-gpu training). Later I found HTA is parsing the communication kernel using: NCCL_KERNEL_RE = re.compile(r"^nccl.*Kernel") in utils.py. However, I found my trace only have nccl::Allreduce as the name after I parse it. Is there a specific parameter I should set to let the pytorch profiler profile the kernel-level detail, or I made some mistake in writing the training script, so the nccl kernel is never being called?
Code
/home/yu/.local/lib/python3.8/site-packages/hta/analyzers/communication_analysis.py:73: RuntimeWarning: invalid value encountered in scalar divide return (shifted_overlap["time_y"] - shifted_overlap["time_x"]).sum() / (
What have you tried?
No response
Environment
Python: 3.10.16 Pytorch: 2.7 nccl/2.19.3-1.awsplugin cuda/11.8.0