Got NaN for communication time, looking for ncclKernel in the trace

What is your question?

I'm using HTA to analyze my tracem which is got from the pytorch profiler. However I got NaN for the communication time, and I dive into the code and I found in the communication_analysis.py, the communication kernel in my trace is 0 (which is not the case since I'm doing multi-gpu training). Later I found HTA is parsing the communication kernel using: NCCL_KERNEL_RE = re.compile(r"^nccl.*Kernel") in utils.py. However, I found my trace only have nccl::Allreduce as the name after I parse it. Is there a specific parameter I should set to let the pytorch profiler profile the kernel-level detail, or I made some mistake in writing the training script, so the nccl kernel is never being called?

Code

/home/yu/.local/lib/python3.8/site-packages/hta/analyzers/communication_analysis.py:73: RuntimeWarning: invalid value encountered in scalar divide return (shifted_overlap["time_y"] - shifted_overlap["time_x"]).sum() / (

What have you tried?

No response

Environment

Python: 3.10.16 Pytorch: 2.7 nccl/2.19.3-1.awsplugin cuda/11.8.0