Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

SeedDatasetFilter(criteria=[]) silently matches no datasets

Open
  • Is something wrong?
#2,947 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@bIackr0se is already working on this.

Since Oct 1, 2026.

  • #2951 by @bIackr0se — open
  • #2967 by @rwinkelman — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
57/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data

Research direction

Start with SeedDatasetFilter.init and _validate() in pyrit/datasets/seed_datasets/seed_metadata.py, then trace _match_filter_to_metadata in seed_dataset_provider.py. Run the reproduction in the issue and add regression coverage for criteria=[]; done means it no longer silently produces an empty result, using one agreed behavior for empty criteria.

Written by the indexing model from the issue text.

Description

Bug: triage
Describe the bug

SeedDatasetFilter(criteria=[]) constructs without error or warning and then matches no dataset, regardless of what is registered.

__init__ takes the criteria is not None branch and stores the empty list as-is:

# pyrit/datasets/seed_datasets/seed_metadata.py
if criteria is not None:
    self.criteria = criteria
elif kwargs:
    self.criteria = [SeedDatasetMetadata(**kwargs)]
else:
    self.criteria = [SeedDatasetMetadata()]

_validate() then has nothing to iterate, and matching reduces to any() over an empty generator:

# pyrit/datasets/seed_datasets/seed_dataset_provider.py
return any(
    cls._match_single_criterion(metadata=metadata, criterion=c, strict_match=dataset_filter.strict_match)
    for c in dataset_filter.criteria
)

so every dataset evaluates to False. The caller gets an empty result list with no indication that the filter itself was the problem — the likely shape being a criteria list assembled dynamically that happens to come out empty.

This is adjacent to, but distinct from, the empty-set-on-an-axis case handled in #2907: that one is about set() on a single axis having opposite meanings under strict_match, whereas this is an empty OR-list across criteria and has no strict_match interaction at all. Noting it here so the two aren't conflated — #2907's description and the docstring on TestStrictMatchSingularFieldValidation::test_strict_match_validates_criteria_list both use the phrase "criteria=[]" loosely to mean "via the criteria-list form," which makes this gap read as already covered when it isn't.

Steps/Code to Reproduce
from pyrit.datasets.seed_datasets.seed_dataset_provider import SeedDatasetProvider
from pyrit.datasets.seed_datasets.seed_metadata import SeedDatasetFilter, SeedDatasetMetadata

f = SeedDatasetFilter(criteria=[])
print(f.criteria)          # []
print(f.has_all_tag)       # False

metadata = SeedDatasetMetadata(tags={"safety"}, size={"small"})
print(SeedDatasetProvider._match_filter_to_metadata(metadata=metadata, dataset_filter=f))
Expected Results

Either:

  1. ValueError at construction, consistent with the other impossible-filter guards already in _validate() (strict_match with multiple values for a singular field, and the empty-axis check added in #2907); or
  2. treat criteria=[] the same as "no criteria supplied" and fall through to [SeedDatasetMetadata()], i.e. an unfiltered match-all filter.

Option 1 looks more consistent with the existing validation, which prefers to fail loudly on a filter that cannot express a meaningful query. Option 2 is defensible if an empty criteria list is considered a legitimate "no constraints" input. Either is preferable to the current silent empty result.

Actual Results

No exception and no warning. f.criteria is [] and _match_filter_to_metadata returns False for every dataset, so get_all_dataset_names_async(filters=f) returns an empty list.

Versions
  • OS: Windows
  • Python version: 3.14.2
  • PyRIT version: 1.2.0.dev0 (installed from main in editable mode)
Dominant language
Python
Stars
4.6k
Forks
924
Avg merge
2d 10h
Merged PRs (30d)
220

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.