SeedDatasetFilter(criteria=[]) silently matches no datasets

Geschlossen
  • Stimmt etwas nicht?
#2,947 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

@bIackr0se arbeitet bereits daran.

Seit 01.10.2026.

  • #2951 von @bIackr0se — offen
  • #2967 von @rwinkelman — offen

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
57/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
data

Rechercherichtung

Start with SeedDatasetFilter.init and _validate() in pyrit/datasets/seed_datasets/seed_metadata.py, then trace _match_filter_to_metadata in seed_dataset_provider.py. Run the reproduction in the issue and add regression coverage for criteria=[]; done means it no longer silently produces an empty result, using one agreed behavior for empty criteria.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Bug: triage
Describe the bug

SeedDatasetFilter(criteria=[]) constructs without error or warning and then matches no dataset, regardless of what is registered.

__init__ takes the criteria is not None branch and stores the empty list as-is:

# pyrit/datasets/seed_datasets/seed_metadata.py
if criteria is not None:
    self.criteria = criteria
elif kwargs:
    self.criteria = [SeedDatasetMetadata(**kwargs)]
else:
    self.criteria = [SeedDatasetMetadata()]

_validate() then has nothing to iterate, and matching reduces to any() over an empty generator:

# pyrit/datasets/seed_datasets/seed_dataset_provider.py
return any(
    cls._match_single_criterion(metadata=metadata, criterion=c, strict_match=dataset_filter.strict_match)
    for c in dataset_filter.criteria
)

so every dataset evaluates to False. The caller gets an empty result list with no indication that the filter itself was the problem — the likely shape being a criteria list assembled dynamically that happens to come out empty.

This is adjacent to, but distinct from, the empty-set-on-an-axis case handled in #2907: that one is about set() on a single axis having opposite meanings under strict_match, whereas this is an empty OR-list across criteria and has no strict_match interaction at all. Noting it here so the two aren't conflated — #2907's description and the docstring on TestStrictMatchSingularFieldValidation::test_strict_match_validates_criteria_list both use the phrase "criteria=[]" loosely to mean "via the criteria-list form," which makes this gap read as already covered when it isn't.

Steps/Code to Reproduce
from pyrit.datasets.seed_datasets.seed_dataset_provider import SeedDatasetProvider
from pyrit.datasets.seed_datasets.seed_metadata import SeedDatasetFilter, SeedDatasetMetadata

f = SeedDatasetFilter(criteria=[])
print(f.criteria)          # []
print(f.has_all_tag)       # False

metadata = SeedDatasetMetadata(tags={"safety"}, size={"small"})
print(SeedDatasetProvider._match_filter_to_metadata(metadata=metadata, dataset_filter=f))
Expected Results

Either:

  1. ValueError at construction, consistent with the other impossible-filter guards already in _validate() (strict_match with multiple values for a singular field, and the empty-axis check added in #2907); or
  2. treat criteria=[] the same as "no criteria supplied" and fall through to [SeedDatasetMetadata()], i.e. an unfiltered match-all filter.

Option 1 looks more consistent with the existing validation, which prefers to fail loudly on a filter that cannot express a meaningful query. Option 2 is defensible if an empty criteria list is considered a legitimate "no constraints" input. Either is preferable to the current silent empty result.

Actual Results

No exception and no warning. f.criteria is [] and _match_filter_to_metadata returns False for every dataset, so get_all_dataset_names_async(filters=f) returns an empty list.

Versions
  • OS: Windows
  • Python version: 3.14.2
  • PyRIT version: 1.2.0.dev0 (installed from main in editable mode)
Vorherrschende Sprache
Python
Sterne
4.7k
Forks
1k
Ø Merge
2 T. 4 Std.
Gemergte PRs (30 T.)
484

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus microsoft/PyRIT

Alle Issues in microsoft/PyRIT

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.