BUG: regex, package-hallucination and aggregator scorer identities collide across configs
维护者通常 2 天内回复
@richlundeen 已经在做这个了。
开始于 2026年10月10日。
评估
调研方向
Start with RegexScorer._build_identifier and the _ConfigurableRegexScorerMixin scorers (XSS, SQLi, SSRF, MarkdownInjection), then compare with CredentialLeakScorer, which already hashes its patterns. Also check PackageHallucinationScorer and the FloatScaleScorerByCategory and FloatScaleScorerAllCategories aggregators, whose __name__ values collide. Reproduce the four collision cases in the issue before changing anything. Done means distinct identifiers for differently configured scorers, default-config hashes unchanged, and a maintainer decision on the metrics registry.
由索引模型根据 Issue 内容生成。
描述
Some scorer identifiers leave out the config that decides the result, so differently configured scorers share a hash and eval_hash. That means get_scorer_metrics() and the evaluator's skip-if-exists can return another config's metrics, and stored scores can't be told apart.
RegexScorer._build_identifieronly recordspattern_count, so any two RegexScorers with the same number of patterns collide. Same for the_ConfigurableRegexScorerMixinscorers (XSS, SQLi, SSRF, MarkdownInjection) with custom patterns. CredentialLeakScorer already hashes its patterns, so it's just the base that's missing it.PackageHallucinationScoreronly recordsknown_package_count.FloatScaleScorerByCategory.MAXandFloatScaleScorerAllCategories.MAXboth have__name__MAX_by_category_(same for MIN/AVERAGE), and scorers recordaggregator.__name__. One returns one score per category (and makes FloatScaleThresholdScorer raise), the other returns one score.
RegexScorer A vs B (1 pattern each, different patterns): same hash True, same eval_hash True; A -> false, B -> true
Markdown default vs custom patterns: same eval_hash True
PackageHallucination {requests} vs {evilpkg}: eval_hash equal True
aggregator names: MAX_by_category_ MAX_by_category_ -> same eval_hash True
A fix would add a pattern/package digest (only when it differs from the defaults, so stored default hashes stay put) and give the all-categories aggregators their own names. Since this moves some hashes, filing first to check how you want the metrics registry handled.
- 主要语言
- Python
- 星标
- 4.6k
- 派生
- 944
- 平均合并
- 2 天 20 小时
- 30 天内合并 PR
- 309
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
microsoft/PyRIT 的其他 Issue
-
BUG UnicodeReplacementConverter writes characters outside the BMP as invalid five-digit \u escapes可能已有人在做 @ShivamSharma43 于 2 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 2 天内回复
-
BUG: label queries only return the final conversation of an attack可能已有人在做 @richlundeen 于 2 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
microsoft/PyRIT#3095 · 1 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
-
BUG Configuration keeps runtime-status errors after polling recovers可能已有人在做 @joeyvictorino 于 2 天前认领。 未关闭Bug: triage GUI help wanted
难度 2/5 1-3 小时 新手友好度 86/100
microsoft/PyRIT#2868 · 4 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
-
BUG JailBreakV-28K loader reads a `transfer_attack_type` column that does not exist upstream and drops the real `from` / `transfer_from_llm` provenance可能已有人在做 @kushaltalati 于 1 天前认领。 未关闭
难度 1/5 1-3 小时 新手友好度 35/100
microsoft/PyRIT#3161 · 1 条评论 ·
维护者通常 2 天内回复
-
BUG QuestionAnswerScorer counts a wrong option as correct when the answer appears inside a longer word or number可能已有人在做 @barkha1research-ai 于 1 天前认领。 未关闭Bug: triage
难度 2/5 1-3 小时 新手友好度 25/100
维护者通常 2 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 72/100
-
难度 1/5 1 小时以内 新手友好度 62/100
-
难度 1/5 1 小时以内 新手友好度 70/100
-
bug
难度 2/5 1-3 小时 新手友好度 72/100
topoteretes/cognee#5773 ·
维护者通常 1 天内回复
-
MiKTeX: a failed build reports no errors because texify --quiet switches the log to file:line: style未关闭
难度 2/5 1-3 小时 新手友好度 74/100
SublimeText/LaTeXTools#1718 ·