derive_metric results are wrong, or aggregation fails, for runs that failed under ignore_failures=True

Aperta
  • Qualcosa non va?
#89 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
54/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
pandas, python
Ambito
data

Direzione di ricerca

Read extract_df_from_data in nsight/extraction.py to follow how failed runs produce derived metric rows, then review the aggregation in nsight/transformation.py. Run the focused tests in tests/test_collection.py and compare failed-run rows with successful derived metrics. Done means failed runs retain the successful metrics’ names and units without a false warning, including the documented all-runs-failed behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Description

When a run fails inside nsight.annotate(..., ignore_failures=True), derive_metric is not called for it (there are no values). The derived row of that run is then recorded as if derive_metric had returned a bare number: the metric is named after the function, the unit is NaN, and a "does not have a unit specified" warning is emitted.

Case A, derive_metric returns a dict and one configuration always fails:

import nsight
import torch

def per_element(time_ns, n):
    return {
        "elements_per_ns": (n * n / time_ns, "elem/ns"),
        "ns_per_element": (time_ns / (n * n), "ns/elem"),
    }

@nsight.analyze.kernel(configs=[(64,), (128,)], runs=2, derive_metric=per_element)
def bench(n):
    a = torch.randn(n, n, device="cuda")
    with nsight.annotate("add", ignore_failures=True):
        if n == 128:
            raise RuntimeError("this config is not supported")
        _ = a + a

print(bench().to_dataframe()[["Annotation", "n", "Metric", "NumRuns", "Unit"]])

UserWarning: Derived metric 'per_element' does not have a unit specified. Return a tuple (value, unit) instead of just the value. ...
Annotation   n                 Metric  NumRuns    Unit
       add  64 gpu__time_duration.sum        2      ns
       add 128 gpu__time_duration.sum        0      ns
       add  64        elements_per_ns        2 elem/ns
       add  64         ns_per_element        2 ns/elem
       add 128            per_element        0     NaN

n=128 has a row for a metric called per_element, and no rows for elements_per_ns / ns_per_element.

Case B, derive_metric returns (value, unit) and only one of several runs of a configuration fails (runs=3, the second run raises):

AssertionError: Column 'Unit' has multiple values in group: <StringArray>
['elem/ns', nan]

The call fails in aggregate_data and returns nothing, although two runs succeeded.

Expected

The failed run counts as a missing value of the same derived metrics, with their units, the way gpu__time_duration.sum is already reported for the failed configuration above. No unit warning when derive_metric returns units. The ignore_failures docstring says "The measured metric for this run will be set to NaN".

Where

extract_df_from_data in nsight/extraction.py: derived_metric is None for a failed run and falls into the branch for a scalar return value. In case B the Unit column then has two values in one group, which the aggregation in nsight/transformation.py rejects.

Environment

nsight-python main at c89d4b5, Python 3.12.3, pandas 3.1.1, Linux x86_64.

How this was checked: no GPU was available, so the outputs above come from the same decorator and annotation calls with the profiling backend replaced by a mock. The mock reports the kernel as fake_kernel, and that column is cut from the table above. The decorator, run loop, extraction and aggregation were the real nsight code. The claim that the real NCU and CUPTI collectors pass no values for a failed run is based on reading extraction.py and collection/cupti.py, not on a GPU run.

Draft PR #74 changes the same function but not this part. A fix is ready if you approve it: it gives failed-run rows the metric names and units of a successful run, and adds two unit tests in tests/test_collection.py. One open point is what to report when every run of a configuration fails, because then there is no successful run to copy names and units from. The fix keeps today's output for that case, without the false warning.

This was found with an AI coding assistant (Claude Code, claude-fable-5-2), which also wrote and ran the mocked repro and drafted this report.

Lingua principale
Python
Stelle
301
Fork
23
Merge medio
2g 20h
PR unite (30g)
12

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/nsight-python

Tutte le issue di NVIDIA/nsight-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.