derive_metric results are wrong, or aggregation fails, for runs that failed under ignore_failures=True

Abierto
  • ¿Algo no está bien?
#89 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
54/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
pandas, python
Área
data

Línea de trabajo

Read extract_df_from_data in nsight/extraction.py to follow how failed runs produce derived metric rows, then review the aggregation in nsight/transformation.py. Run the focused tests in tests/test_collection.py and compare failed-run rows with successful derived metrics. Done means failed runs retain the successful metrics’ names and units without a false warning, including the documented all-runs-failed behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Description

When a run fails inside nsight.annotate(..., ignore_failures=True), derive_metric is not called for it (there are no values). The derived row of that run is then recorded as if derive_metric had returned a bare number: the metric is named after the function, the unit is NaN, and a "does not have a unit specified" warning is emitted.

Case A, derive_metric returns a dict and one configuration always fails:

import nsight
import torch

def per_element(time_ns, n):
    return {
        "elements_per_ns": (n * n / time_ns, "elem/ns"),
        "ns_per_element": (time_ns / (n * n), "ns/elem"),
    }

@nsight.analyze.kernel(configs=[(64,), (128,)], runs=2, derive_metric=per_element)
def bench(n):
    a = torch.randn(n, n, device="cuda")
    with nsight.annotate("add", ignore_failures=True):
        if n == 128:
            raise RuntimeError("this config is not supported")
        _ = a + a

print(bench().to_dataframe()[["Annotation", "n", "Metric", "NumRuns", "Unit"]])

UserWarning: Derived metric 'per_element' does not have a unit specified. Return a tuple (value, unit) instead of just the value. ...
Annotation   n                 Metric  NumRuns    Unit
       add  64 gpu__time_duration.sum        2      ns
       add 128 gpu__time_duration.sum        0      ns
       add  64        elements_per_ns        2 elem/ns
       add  64         ns_per_element        2 ns/elem
       add 128            per_element        0     NaN

n=128 has a row for a metric called per_element, and no rows for elements_per_ns / ns_per_element.

Case B, derive_metric returns (value, unit) and only one of several runs of a configuration fails (runs=3, the second run raises):

AssertionError: Column 'Unit' has multiple values in group: <StringArray>
['elem/ns', nan]

The call fails in aggregate_data and returns nothing, although two runs succeeded.

Expected

The failed run counts as a missing value of the same derived metrics, with their units, the way gpu__time_duration.sum is already reported for the failed configuration above. No unit warning when derive_metric returns units. The ignore_failures docstring says "The measured metric for this run will be set to NaN".

Where

extract_df_from_data in nsight/extraction.py: derived_metric is None for a failed run and falls into the branch for a scalar return value. In case B the Unit column then has two values in one group, which the aggregation in nsight/transformation.py rejects.

Environment

nsight-python main at c89d4b5, Python 3.12.3, pandas 3.1.1, Linux x86_64.

How this was checked: no GPU was available, so the outputs above come from the same decorator and annotation calls with the profiling backend replaced by a mock. The mock reports the kernel as fake_kernel, and that column is cut from the table above. The decorator, run loop, extraction and aggregation were the real nsight code. The claim that the real NCU and CUPTI collectors pass no values for a failed run is based on reading extraction.py and collection/cupti.py, not on a GPU run.

Draft PR #74 changes the same function but not this part. A fix is ready if you approve it: it gives failed-run rows the metric names and units of a successful run, and adds two unit tests in tests/test_collection.py. One open point is what to report when every run of a configuration fails, because then there is no successful run to copy names and units from. The fix keeps today's output for that case, without the false warning.

This was found with an AI coding assistant (Claude Code, claude-fable-5-2), which also wrote and ran the mocked repro and drafted this report.

Lenguaje dominante
Python
Estrellas
301
Forks
23
Merge medio
2 d 20 h
PR fusionados (30 d)
12

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/nsight-python

Todos los issues de NVIDIA/nsight-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.