Wire doc/scanner notebooks into the integration notebook harness

Open
#1,752 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
jupyter-notebook, python
Domain
testing-qa

Research direction

Start with tests/integration/scenarios/test_notebooks_scenarios.py, pyrit.common.path.DOCS_CODE_PATH, and the notebooks under doc/scanner/. Compare the existing per-area harness and determine how max_dataset_size=1 or skipped_files should handle the slow notebooks. Done means the scanner notebooks execute under RUN_ALL_TESTS=true and their user-facing API paths are covered by CI.

Written by the indexing model from the issue text.

Description

not ready yet

Problem

The doc/scanner/ notebooks (1_pyrit_scan.ipynb, airt.ipynb, benchmark.ipynb, foundry.ipynb, garak.ipynb) are listed in doc/myst.yml and render in our docs site, but nothing in CI actually executes them. The integration notebook harness (tests/integration/<area>/test_notebooks_*.py) is hard-coded to doc/code/<area>/ via pyrit.common.path.DOCS_CODE_PATH, so the scanner notebooks are render-only.

This means breakages in the user-facing scanner API surface — like the deprecated-type slip-through fixed in #1746 — won't be caught by CI.

Proposal

Add tests/integration/scanner/test_notebooks_scanner.py, mirroring the existing per-area pattern (e.g. tests/integration/scenarios/test_notebooks_scenarios.py). It would parametrize over os.listdir(DOC_ROOT / "scanner") and run each notebook via ExecutePreprocessor under RUN_ALL_TESTS=true.

Points to confirm during implementation:

  • The scanner notebooks call pyrit_scan end-to-end with OpenAIChatTarget() against real datasets. They are likely slow enough to warrant max_dataset_size=1 shims or a skipped_files entry for the heavy ones — pick the right cost/coverage tradeoff up front.
  • DOCS_CODE_PATH is doc/code by name; introducing a parallel DOCS_SCANNER_PATH constant in pyrit/common/path.py keeps the pattern consistent.

Context

Surfaced in the PR review thread on #1746: https://github.com/microsoft/PyRIT/pull/1746#issuecomment-4478807681

Dominant language
Python
Stars
4.5k
Forks
896
Avg merge
3d 7h
Merged PRs (30d)
155

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.