Is HTA working??

What is your question?

Hi,

Would appreciate any help here.

I am learning about HTA, and just following some basic tutorials to get me started. Specifically, https://pytorch.org/tutorials/beginner/hta_intro_tutorial.html

I've built a very simple toy example, ran the profiler and dumped the chrome traces into a file. Manually augmented the trace with a rank (since my toy example is single threaded).

This is a snippet of my toy example code where I used the profiler.

tracing_schedule = schedule(skip_first=3, wait=0, warmup=2, active=45, repeat=1) trace_handler = tensorboard_trace_handler(dir_name='./logs') mycls = MyClass1()

with profile( activities = [ProfilerActivity.CPU, ProfilerActivity.CUDA], schedule = tracing_schedule, on_trace_ready = trace_handler, profile_memory = True, record_shapes = True, with_stack = True ) as prof: for i in range(100): mycls.random_training() prof.step()

Then in Juypter Notebook, I can load up the trace file fine. from hta.trace_analysis import TraceAnalysis trace_dir = "./logs" analyzer = TraceAnalysis(trace_dir=trace_dir)

2025-03-13 08:47:23,390 - hta - trace.py:L389 - INFO - /home/dtu1/pyprojects/pyplayground/logs 2025-03-13 08:47:23,391 - hta - trace_file.py:L94 - INFO - Rank to trace file map: {0: '/home/dtu1/pyprojects/pyplayground/logs/HERMLSGPU021NLD_46619.1741841940896462986.pt.trace.json'} 2025-03-13 08:47:23,392 - hta - trace.py:L535 - INFO - ranks=[0] 2025-03-13 08:47:23,507 - hta - trace.py:L118 - INFO - Parsed /home/dtu1/pyprojects/pyplayground/logs/HERMLSGPU021NLD_46619.1741841940896462986.pt.trace.json time = 0.11 seconds

However, no matter what I try with the analyzer, I get errors. Haven't got a single analyzer method to work without errors or showing me anything. time_spent_df = analyzer.get_temporal_breakdown() idle_time_df = analyzer.get_idle_time_breakdown() kernel_type_metrics_df, kernel_metrics_df = analyzer.get_gpu_kernel_breakdown() overlap_df = analyzer.get_comm_comp_overlap()

For example, when I run get_temporal_breakdown(), I get this IndexError

---

IndexError Traceback (most recent call last) Cell In[8], line 1 ----> 1 time_spent_df = analyzer.get_temporal_breakdown()

File ~/pyprojects/pyenvs/drlsched-env2/lib/python3.10/site-packages/hta/trace_analysis.py:171, in TraceAnalysis.get_temporal_breakdown(self, visualize) 155 def get_temporal_breakdown(self, visualize: bool = True) -> pd.DataFrame: 156 r""" 157 Compute the idle time, compute time and non-compute time for each rank. Time is measured in 158 nanoseconds (ns). non-compute time is defined as the total time the GPU is not executing a (...) 169 time for each rank. 170 """ --> 171 return BreakdownAnalysis.get_temporal_breakdown(self.t, visualize)

File ~/pyprojects/pyenvs/drlsched-env2/lib/python3.10/site-packages/hta/analyzers/breakdown_analysis.py:362, in BreakdownAnalysis.get_temporal_breakdown(cls, t, visualize) 360 for rank, trace_df in t.traces.items(): 361 result["rank"].append(rank) --> 362 idle_time, compute_time, non_compute_time, kernel_time = idle_time_per_rank( 363 trace_df 364 ) 365 result["idle_time(us)"].append(idle_time) 366 result["compute_time(us)"].append(compute_time)

File ~/pyprojects/pyenvs/drlsched-env2/lib/python3.10/site-packages/hta/analyzers/breakdown_analysis.py:338, in BreakdownAnalysis.get_temporal_breakdown..idle_time_per_rank(trace_df) 336 """returns idle_time (us) , compute_time (us), non_compute_time (us), total_time (us)""" 337 gpu_kernels = trace_df[trace_df["stream"].ne(-1)].copy() --> 338 idle_time, kernel_time = cls._get_idle_time_for_kernels(gpu_kernels) 340 gpu_kernels["kernel_type"] = gpu_kernels[["name"]].apply( 341 lambda x: get_kernel_type(sym_table[x["name"]]), axis=1 342 ) 344 # Isolate computation kernels and merge each one of them.

File ~/pyprojects/pyenvs/drlsched-env2/lib/python3.10/site-packages/hta/analyzers/breakdown_analysis.py:323, in BreakdownAnalysis._get_idle_time_for_kernels(cls, kernels_df) 313 """ 314 Compute idle time for given set of GPU kernels : 315 returns : (...) 320 so total time is exclusive of that. 321 """ 322 merged_kernels = merge_kernel_intervals(kernels_df) --> 323 kernel_time = merged_kernels.iloc[-1]["end"] - merged_kernels.iloc[0]["ts"] 324 # differences of end - ts are commutative 325 kernel_run_time = merged_kernels.end.sum() - merged_kernels.ts.sum()

File ~/pyprojects/pyenvs/drlsched-env2/lib/python3.10/site-packages/pandas/core/indexing.py:1103, in _LocationIndexer.getitem(self, key) 1100 axis = self.axis or 0 1102 maybe_callable = com.apply_if_callable(key, self.obj) -> 1103 return self._getitem_axis(maybe_callable, axis=axis)

File ~/pyprojects/pyenvs/drlsched-env2/lib/python3.10/site-packages/pandas/core/indexing.py:1656, in _iLocIndexer._getitem_axis(self, key, axis) 1653 raise TypeError("Cannot index by location index with a non-integer key") 1655 # validate the location -> 1656 self._validate_integer(key, axis) 1658 return self.obj._ixs(key, axis=axis)

File ~/pyprojects/pyenvs/drlsched-env2/lib/python3.10/site-packages/pandas/core/indexing.py:1589, in _iLocIndexer._validate_integer(self, key, axis) 1587 len_axis = len(self.obj._get_axis(axis)) 1588 if key >= len_axis or key < -len_axis: -> 1589 raise IndexError("single positional indexer is out-of-bounds")

IndexError: single positional indexer is out-of-bounds

I thought it might be related to my trace files, but I appear to be able to open the trace files fine in chrome://tracing

Any insights / guideance appreciated!

Regards, Dean

Code

No response

What have you tried?

No response

Environment

Installed HTA using pip install from the pip package.

Also tried directly using the source from git.

Both had the same issues.