Report harm scorer accuracy split by inter-rater agreement (a third of violence gold labels are single-vote)

Start with HarmScorerEvaluator and the multi-rater datasets under pyrit/datasets/scorer_evals/harm. Read pyrit.score.scorer_evaluation.krippendorff.krippendorff_alpha and the existing *_metrics.jsonl output to understand agreement handling. Done means preserving current aggregate metrics while reporting accuracy and MAE separately for unanimous and contested rows, with any single-rater metadata clearly marked if included.

Is your feature request related to a problem?

A harm scorer is evaluated against one gold label per response. Where that label is the mean of several human raters, the mean hides how much the raters disagreed, and the eval reports a single MAE/accuracy as if the label were certain. On the multi-rater gold sets in pyrit/datasets/scorer_evals/harm, a large fraction of the binary harmful/not labels come down to a single rater's vote.

Measured on the current files (Krippendorff alpha computed with PyRIT's own pyrit.score.scorer_evaluation.krippendorff.krippendorff_alpha, ordinal, so it matches the *_metrics.jsonl values):

On violence, nearly a third of the binary gold labels are 2-1 splits, so a scorer marked wrong on those rows was outvoted by one human rather than overruled by a consensus. A point metric spends the scorer's error budget partly on rows the humans themselves split on, and a scorer that is well calibrated to that ambiguity (for example one that abstains where humans divide) is penalised for it rather than credited.

The other eight harm gold sets (exploits, hate_speech, info_integrity, privacy, representational, self_harm, sexual, violence) ship a single rating, so their disagreement cannot be observed at all and their labels are treated as ground truth with no uncertainty signal.

Describe the solution you'd like

1. In HarmScorerEvaluator, when a gold set has more than one rater, additionally report accuracy/MAE split by agreement level: unanimous rows vs contested (non-unanimous) rows, from the same eval run. A scorer at 0.87 overall that is 0.95 on agreed rows and near chance on contested ones is behaving correctly, and the split makes that visible where the aggregate hides it.
2. 2. Optionally mark the single-rater gold sets as provisional in the harm-definition metadata, so a low scorer number on them is read as possible label uncertainty rather than a scorer defect.

Describe alternatives you've considered

Down-weighting contested rows in the aggregate metric, or dropping them. Reporting the split is less opinionated: it changes nothing about the existing numbers and adds a second view, so no current metric moves.

Additional context

I have a small standalone audit script (numpy plus PyRIT's Krippendorff) that produces the table above and emits the contested row indices per file, which is what recommendation 1 would filter on. Happy to attach it or open a PR for the reporting change if this direction is wanted. Flagging @romanlutz since this feeds directly into the dataset-curation work.