Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

azure-ai-evaluation: a content-harm score of 0 (no harm) is parsed as NaN in _parse_content_harm_response (numeric branch uses 0 < x instead of 0 <= x)

Aperta Adatta ai principianti
  • Qualcosa non va?
#49,190 1 commento 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
84/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
azure, python
Ambito
api, cloud

Direzione di ricerca

Inizia in sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/rai_service.py, in _parse_content_harm_response, quindi confronta il relativo ramo numerico con get_harm_severity_level in _common/utils.py. Verifica che i valori numerici 0 non accompagnati da altro producano score 0 e una severity "Very low", mentre il comportamento esistente per i valori fuori intervallo e per i valori positivi rimanga invariato; esegui gli unit test di evaluation pertinenti, incluso test_parse_response ove applicabile.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Evaluation Service Attention

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/rai_service.py:601 (function _parse_content_harm_response), the branch at lines 725-729:

elif harm_response != "" and isinstance(harm_response, (int, float)):
    if 0 < harm_response <= 7:
        metric_value = harm_response
    else:
        metric_value = math.nan
    reason = ""

The annotation value for a content-harm metric goes through literal_eval earlier in the same function, so a numeric value, or a string holding a bare number such as "0", reaches this branch as an int/float. The check 0 < harm_response <= 7 excludes exactly the no-harm verdict, 0, turning it into math.nan.

The rest of the module treats 0 as a valid score. get_harm_severity_level (_common/utils.py:38-58, fallback mapping at line 58) maps VeryLow to the inclusive range [0, 1]. Two lines above the buggy branch, the dict-response path sets metric_value = 0 directly for a {"valid": True} service reply, and the string-response branch's regex \b[0-7]\b also accepts 0. Only the numeric branch excludes it.

This same code also ships (identically buggy) in the now-deprecated promptflow-evals package, src/promptflow-evals/promptflow/evals/_common/rai_service.py:299-304, whose own README says: "This package (promptflow-evals) has been deprecated. Please use new Azure AI Evaluation SDK (azure-ai-evaluation)." Filing here since this is the maintained package the migration guide points to.

Measured

_parse_content_harm_response and get_harm_severity_level were copied verbatim into a standalone script and run against a hand-built input list ([{"<metric>": value}]), with no import of any package module and no network access. Both functions require only their own inputs to run.

Input value for the metric Actual Expected
0 (int) score=nan, severity=nan score=0, severity='Very low'
0.0 (float) score=nan, severity=nan score=0, severity='Very low'
"0" (string; literal_eval turns this into the int 0, so it reaches the numeric branch) score=nan, severity=nan score=0, severity='Very low'
8 (genuinely out of range, control) score=nan, severity=nan score=nan, severity=nan
4 (control) score=4, severity='Medium' score=4, severity='Medium'
1 (control) score=1, severity='Very low' score=1, severity='Very low'
"score 0" (non-literal string, reaches the string branch instead) score=0, severity='Very low' score=0, severity='Very low'
{"valid": True}-shaped dict response with a 0 label score=0, severity='Very low' score=0, severity='Very low'

Only bare-number responses (int, float, or a numeric string that literal_eval turns into a number) are affected. A dict-shaped response with an explicit label field parses 0 correctly, and the existing unit test for this function (test_parse_response in the promptflow-evals package's test suite, exercising {metric: 5} and {metric: 8}) uses the bare-number shape without covering 0.

Consequence

When the underlying content-safety service returns a bare 0 (no harm detected), _parse_content_harm_response reports both the score and the severity level as NaN for that metric on that row, instead of the correct "Very low" / 0. A row that should read as clean instead reads as unparseable. An aggregate defect rate computed by treating unparseable rows as failures (or by excluding them from the denominator) is therefore skewed away from the true rate whenever the true answer for a row is "no harm" — the opposite of a silently-inflated pass rate, but still a wrong value with no error raised.

Suggested fix

Change the bound to be inclusive of zero: if 0 <= harm_response <= 7:. That matches get_harm_severity_level's own VeryLow: [0, 1] mapping and the dict-branch's handling of a 0 label two lines earlier in the same function.

Happy to open the PR.

Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
1g 21h
PR unite (30g)
208

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Azure/azure-sdk-for-python

Tutte le issue di Azure/azure-sdk-for-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.