derive_metric results are wrong, or aggregation fails, for runs that failed under ignore_failures=True

Offen
  • Stimmt etwas nicht?
#89 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
54/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
pandas, python
Bereich
data

Rechercherichtung

Read extract_df_from_data in nsight/extraction.py to follow how failed runs produce derived metric rows, then review the aggregation in nsight/transformation.py. Run the focused tests in tests/test_collection.py and compare failed-run rows with successful derived metrics. Done means failed runs retain the successful metrics’ names and units without a false warning, including the documented all-runs-failed behavior.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Description

When a run fails inside nsight.annotate(..., ignore_failures=True), derive_metric is not called for it (there are no values). The derived row of that run is then recorded as if derive_metric had returned a bare number: the metric is named after the function, the unit is NaN, and a "does not have a unit specified" warning is emitted.

Case A, derive_metric returns a dict and one configuration always fails:

import nsight
import torch

def per_element(time_ns, n):
    return {
        "elements_per_ns": (n * n / time_ns, "elem/ns"),
        "ns_per_element": (time_ns / (n * n), "ns/elem"),
    }

@nsight.analyze.kernel(configs=[(64,), (128,)], runs=2, derive_metric=per_element)
def bench(n):
    a = torch.randn(n, n, device="cuda")
    with nsight.annotate("add", ignore_failures=True):
        if n == 128:
            raise RuntimeError("this config is not supported")
        _ = a + a

print(bench().to_dataframe()[["Annotation", "n", "Metric", "NumRuns", "Unit"]])

UserWarning: Derived metric 'per_element' does not have a unit specified. Return a tuple (value, unit) instead of just the value. ...
Annotation   n                 Metric  NumRuns    Unit
       add  64 gpu__time_duration.sum        2      ns
       add 128 gpu__time_duration.sum        0      ns
       add  64        elements_per_ns        2 elem/ns
       add  64         ns_per_element        2 ns/elem
       add 128            per_element        0     NaN

n=128 has a row for a metric called per_element, and no rows for elements_per_ns / ns_per_element.

Case B, derive_metric returns (value, unit) and only one of several runs of a configuration fails (runs=3, the second run raises):

AssertionError: Column 'Unit' has multiple values in group: <StringArray>
['elem/ns', nan]

The call fails in aggregate_data and returns nothing, although two runs succeeded.

Expected

The failed run counts as a missing value of the same derived metrics, with their units, the way gpu__time_duration.sum is already reported for the failed configuration above. No unit warning when derive_metric returns units. The ignore_failures docstring says "The measured metric for this run will be set to NaN".

Where

extract_df_from_data in nsight/extraction.py: derived_metric is None for a failed run and falls into the branch for a scalar return value. In case B the Unit column then has two values in one group, which the aggregation in nsight/transformation.py rejects.

Environment

nsight-python main at c89d4b5, Python 3.12.3, pandas 3.1.1, Linux x86_64.

How this was checked: no GPU was available, so the outputs above come from the same decorator and annotation calls with the profiling backend replaced by a mock. The mock reports the kernel as fake_kernel, and that column is cut from the table above. The decorator, run loop, extraction and aggregation were the real nsight code. The claim that the real NCU and CUPTI collectors pass no values for a failed run is based on reading extraction.py and collection/cupti.py, not on a GPU run.

Draft PR #74 changes the same function but not this part. A fix is ready if you approve it: it gives failed-run rows the metric names and units of a successful run, and adds two unit tests in tests/test_collection.py. One open point is what to report when every run of a configuration fails, because then there is no successful run to copy names and units from. The fix keeps today's output for that case, without the false warning.

This was found with an AI coding assistant (Claude Code, claude-fable-5-2), which also wrote and ran the mocked repro and drafted this report.

Vorherrschende Sprache
Python
Sterne
301
Forks
23
Ø Merge
2 T. 20 Std.
Gemergte PRs (30 T.)
12

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus NVIDIA/nsight-python

Alle Issues in NVIDIA/nsight-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.