facebookresearch/HolisticTraceAnalysis

Got NaN for communication time, looking for ncclKernel in the trace

Open

#247 opened on May 8, 2025

View on GitHubContributor guide
 (1 comment) (0 reactions) (0 assignees)Python (94 forks)auto 404
help wantedneeds triagequestion

Repository metrics

Stars
 (535 stars)
PR merge metrics
 (PR metrics pending)

Description

What is your question?

I'm using HTA to analyze my tracem which is got from the pytorch profiler. However I got NaN for the communication time, and I dive into the code and I found in the communication_analysis.py, the communication kernel in my trace is 0 (which is not the case since I'm doing multi-gpu training). Later I found HTA is parsing the communication kernel using: NCCL_KERNEL_RE = re.compile(r"^nccl.*Kernel") in utils.py. However, I found my trace only have nccl::Allreduce as the name after I parse it. Is there a specific parameter I should set to let the pytorch profiler profile the kernel-level detail, or I made some mistake in writing the training script, so the nccl kernel is never being called?

Code

/home/yu/.local/lib/python3.8/site-packages/hta/analyzers/communication_analysis.py:73: RuntimeWarning: invalid value encountered in scalar divide return (shifted_overlap["time_y"] - shifted_overlap["time_x"]).sum() / (

What have you tried?

No response

Environment

Python: 3.10.16 Pytorch: 2.7 nccl/2.19.3-1.awsplugin cuda/11.8.0

Contributor guide

Research direction
Check if the PyTorch profiler is configured to emit kernel level events (e.g., with `record shapes=True` or `profile memory=True`). Update NCCL KERNEL RE in utils.py to match patterns like `nccl::Allreduce` or confirm that the trace contains kernel level names.
Tech stack
pythonpytorch
Domain
backendmachine learning
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonPyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50