Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

azure-ai-evaluation: a content-harm score of 0 (no harm) is parsed as NaN in _parse_content_harm_response (numeric branch uses 0 < x instead of 0 <= x)

Abierto Apto para principiantes
  • ¿Algo no está bien?
#49,190 1 comentario 1 reacción 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
84/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
azure, python
Área
api, cloud

Línea de trabajo

Comienza en sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/rai_service.py, en _parse_content_harm_response, y compara después su rama numérica con get_harm_severity_level en _common/utils.py. Verifica que los valores numéricos 0 sin más produzcan score 0 y una severidad "Very low", mientras que el comportamiento existente para valores fuera de rango y valores positivos permanezca sin cambios; ejecuta las pruebas unitarias de evaluación relevantes, incluida test_parse_response cuando corresponda.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Evaluation Service Attention

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/rai_service.py:601 (function _parse_content_harm_response), the branch at lines 725-729:

elif harm_response != "" and isinstance(harm_response, (int, float)):
    if 0 < harm_response <= 7:
        metric_value = harm_response
    else:
        metric_value = math.nan
    reason = ""

The annotation value for a content-harm metric goes through literal_eval earlier in the same function, so a numeric value, or a string holding a bare number such as "0", reaches this branch as an int/float. The check 0 < harm_response <= 7 excludes exactly the no-harm verdict, 0, turning it into math.nan.

The rest of the module treats 0 as a valid score. get_harm_severity_level (_common/utils.py:38-58, fallback mapping at line 58) maps VeryLow to the inclusive range [0, 1]. Two lines above the buggy branch, the dict-response path sets metric_value = 0 directly for a {"valid": True} service reply, and the string-response branch's regex \b[0-7]\b also accepts 0. Only the numeric branch excludes it.

This same code also ships (identically buggy) in the now-deprecated promptflow-evals package, src/promptflow-evals/promptflow/evals/_common/rai_service.py:299-304, whose own README says: "This package (promptflow-evals) has been deprecated. Please use new Azure AI Evaluation SDK (azure-ai-evaluation)." Filing here since this is the maintained package the migration guide points to.

Measured

_parse_content_harm_response and get_harm_severity_level were copied verbatim into a standalone script and run against a hand-built input list ([{"<metric>": value}]), with no import of any package module and no network access. Both functions require only their own inputs to run.

Input value for the metric Actual Expected
0 (int) score=nan, severity=nan score=0, severity='Very low'
0.0 (float) score=nan, severity=nan score=0, severity='Very low'
"0" (string; literal_eval turns this into the int 0, so it reaches the numeric branch) score=nan, severity=nan score=0, severity='Very low'
8 (genuinely out of range, control) score=nan, severity=nan score=nan, severity=nan
4 (control) score=4, severity='Medium' score=4, severity='Medium'
1 (control) score=1, severity='Very low' score=1, severity='Very low'
"score 0" (non-literal string, reaches the string branch instead) score=0, severity='Very low' score=0, severity='Very low'
{"valid": True}-shaped dict response with a 0 label score=0, severity='Very low' score=0, severity='Very low'

Only bare-number responses (int, float, or a numeric string that literal_eval turns into a number) are affected. A dict-shaped response with an explicit label field parses 0 correctly, and the existing unit test for this function (test_parse_response in the promptflow-evals package's test suite, exercising {metric: 5} and {metric: 8}) uses the bare-number shape without covering 0.

Consequence

When the underlying content-safety service returns a bare 0 (no harm detected), _parse_content_harm_response reports both the score and the severity level as NaN for that metric on that row, instead of the correct "Very low" / 0. A row that should read as clean instead reads as unparseable. An aggregate defect rate computed by treating unparseable rows as failures (or by excluding them from the denominator) is therefore skewed away from the true rate whenever the true answer for a row is "no harm" — the opposite of a silently-inflated pass rate, but still a wrong value with no error raised.

Suggested fix

Change the bound to be inclusive of zero: if 0 <= harm_response <= 7:. That matches get_harm_severity_level's own VeryLow: [0, 1] mapping and the dict-branch's handling of a 0 label two lines earlier in the same function.

Happy to open the PR.

Lenguaje dominante
Python
Estrellas
5.6k
Forks
3.4k
Merge medio
1 d 21 h
PR fusionados (30 d)
208

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Azure/azure-sdk-for-python

Todos los issues de Azure/azure-sdk-for-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.