Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

BUG: regex, package-hallucination and aggregator scorer identities collide across configs

Open
  • Is something wrong?
#3,113 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 2 days

@richlundeen is already working on this.

Since Oct 10, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
security

Research direction

Start with RegexScorer._build_identifier and the _ConfigurableRegexScorerMixin scorers (XSS, SQLi, SSRF, MarkdownInjection), then compare with CredentialLeakScorer, which already hashes its patterns. Also check PackageHallucinationScorer and the FloatScaleScorerByCategory and FloatScaleScorerAllCategories aggregators, whose __name__ values collide. Reproduce the four collision cases in the issue before changing anything. Done means distinct identifiers for differently configured scorers, default-config hashes unchanged, and a maintainer decision on the metrics registry.

Written by the indexing model from the issue text.

Description

Some scorer identifiers leave out the config that decides the result, so differently configured scorers share a hash and eval_hash. That means get_scorer_metrics() and the evaluator's skip-if-exists can return another config's metrics, and stored scores can't be told apart.

  • RegexScorer._build_identifier only records pattern_count, so any two RegexScorers with the same number of patterns collide. Same for the _ConfigurableRegexScorerMixin scorers (XSS, SQLi, SSRF, MarkdownInjection) with custom patterns. CredentialLeakScorer already hashes its patterns, so it's just the base that's missing it.
  • PackageHallucinationScorer only records known_package_count.
  • FloatScaleScorerByCategory.MAX and FloatScaleScorerAllCategories.MAX both have __name__ MAX_by_category_ (same for MIN/AVERAGE), and scorers record aggregator.__name__. One returns one score per category (and makes FloatScaleThresholdScorer raise), the other returns one score.
RegexScorer A vs B (1 pattern each, different patterns): same hash True, same eval_hash True; A -> false, B -> true
Markdown default vs custom patterns: same eval_hash True
PackageHallucination {requests} vs {evilpkg}: eval_hash equal True
aggregator names: MAX_by_category_ MAX_by_category_ -> same eval_hash True

A fix would add a pattern/package digest (only when it differs from the defaults, so stored default hashes stay put) and give the all-categories aggregators their own names. Since this moves some hashes, filing first to check how you want the metrics registry handled.

Dominant language
Python
Stars
4.6k
Forks
944
Avg merge
3d 1h
Merged PRs (30d)
278

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.