derive_metric results are wrong, or aggregation fails, for runs that failed under ignore_failures=True

Ouverte
  • Quelque chose ne va pas ?
#89 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
54/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
pandas, python
Domaine
data

Piste de recherche

Read extract_df_from_data in nsight/extraction.py to follow how failed runs produce derived metric rows, then review the aggregation in nsight/transformation.py. Run the focused tests in tests/test_collection.py and compare failed-run rows with successful derived metrics. Done means failed runs retain the successful metrics’ names and units without a false warning, including the documented all-runs-failed behavior.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Description

When a run fails inside nsight.annotate(..., ignore_failures=True), derive_metric is not called for it (there are no values). The derived row of that run is then recorded as if derive_metric had returned a bare number: the metric is named after the function, the unit is NaN, and a "does not have a unit specified" warning is emitted.

Case A, derive_metric returns a dict and one configuration always fails:

import nsight
import torch

def per_element(time_ns, n):
    return {
        "elements_per_ns": (n * n / time_ns, "elem/ns"),
        "ns_per_element": (time_ns / (n * n), "ns/elem"),
    }

@nsight.analyze.kernel(configs=[(64,), (128,)], runs=2, derive_metric=per_element)
def bench(n):
    a = torch.randn(n, n, device="cuda")
    with nsight.annotate("add", ignore_failures=True):
        if n == 128:
            raise RuntimeError("this config is not supported")
        _ = a + a

print(bench().to_dataframe()[["Annotation", "n", "Metric", "NumRuns", "Unit"]])

UserWarning: Derived metric 'per_element' does not have a unit specified. Return a tuple (value, unit) instead of just the value. ...
Annotation   n                 Metric  NumRuns    Unit
       add  64 gpu__time_duration.sum        2      ns
       add 128 gpu__time_duration.sum        0      ns
       add  64        elements_per_ns        2 elem/ns
       add  64         ns_per_element        2 ns/elem
       add 128            per_element        0     NaN

n=128 has a row for a metric called per_element, and no rows for elements_per_ns / ns_per_element.

Case B, derive_metric returns (value, unit) and only one of several runs of a configuration fails (runs=3, the second run raises):

AssertionError: Column 'Unit' has multiple values in group: <StringArray>
['elem/ns', nan]

The call fails in aggregate_data and returns nothing, although two runs succeeded.

Expected

The failed run counts as a missing value of the same derived metrics, with their units, the way gpu__time_duration.sum is already reported for the failed configuration above. No unit warning when derive_metric returns units. The ignore_failures docstring says "The measured metric for this run will be set to NaN".

Where

extract_df_from_data in nsight/extraction.py: derived_metric is None for a failed run and falls into the branch for a scalar return value. In case B the Unit column then has two values in one group, which the aggregation in nsight/transformation.py rejects.

Environment

nsight-python main at c89d4b5, Python 3.12.3, pandas 3.1.1, Linux x86_64.

How this was checked: no GPU was available, so the outputs above come from the same decorator and annotation calls with the profiling backend replaced by a mock. The mock reports the kernel as fake_kernel, and that column is cut from the table above. The decorator, run loop, extraction and aggregation were the real nsight code. The claim that the real NCU and CUPTI collectors pass no values for a failed run is based on reading extraction.py and collection/cupti.py, not on a GPU run.

Draft PR #74 changes the same function but not this part. A fix is ready if you approve it: it gives failed-run rows the metric names and units of a successful run, and adds two unit tests in tests/test_collection.py. One open point is what to report when every run of a configuration fails, because then there is no successful run to copy names and units from. The fix keeps today's output for that case, without the false warning.

This was found with an AI coding assistant (Claude Code, claude-fable-5-2), which also wrote and ran the mocked repro and drafted this report.

Langage dominant
Python
Étoiles
301
Forks
23
Merge moyen
2 j 20 h
PR mergées (30 j)
12

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/nsight-python

Toutes les issues de NVIDIA/nsight-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.