BUG MOSSBench unit-test setup fails with an existing SQLite table

Start in tests/unit/datasets/test_mossbench_dataset.py and compare its setup_memory fixture with sqlite_instance and patch_central_database in tests/unit/conftest.py. Run the default-extras Python 3.11 unit-test environment with the four-worker make unit-test-junit command, then isolate the database, results-directory, singleton, and engine lifecycle involved. Done means deterministic per-test isolation and cleanup coverage, with the affected tests verified.

Describe the bug

A main-branch Python unit-test job failed while setting up TestMossBenchDataset.test_dataset_name because SQLite tried to create a table that already existed. The overall workflow was cancelled, so looking only at the run conclusion hides this failed job.

The MOSSBench class has its own autouse setup_memory fixture in tests/unit/datasets/test_mossbench_dataset.py. It constructs default, file-backed SQLiteMemory() and only clears the CentralMemory pointer at teardown. It bypasses the repository's isolated sqlite_instance / patch_central_database fixtures in tests/unit/conftest.py, which scope the database, results directory, central-memory/singleton state, and engine disposal.

This is a concrete isolation concern, but the other database creator and exact remote collision mechanism have not been identified. Please do not treat concurrency as a proven cause or label this a confirmed flake based on the available runs alone.

Related merged work: #2640 introduced isolated SQLite fixture machinery; this custom fixture still bypasses it. No matching open fix or issue was found when filing.

Steps/Code to Reproduce

Observed failure: main run 35668881112, attempt 1, job 106560764233, matrix ubuntu-latest / Python 3.11 / pyrit / dev.

The relevant existing fixture is:

@pytest.fixture(autouse=True)
def setup_memory(self):
    memory = SQLiteMemory()
    CentralMemory.set_memory_instance(memory)
    yield
    CentralMemory.set_memory_instance(None)

This excerpt explains the suspect lifecycle; it is not a standalone deterministic reproducer. Start by reproducing the default-extras Ubuntu/Python 3.11 unit-test environment and the existing four-worker make unit-test-junit command. Establish the database/singleton state and any interacting fixture before changing behavior. A targeted test by itself may pass.

The exact same SHA and matrix cell passed merge-queue run 35667467814, job 106556307066. That was a separate earlier run, not a rerun of the failed main attempt. The audit's later Windows/all-extras suite also passed, which does not reproduce the failed matrix cell.

Expected Results

MOSSBench unit tests should use isolated test-owned memory and must not depend on a persistent default database or leftover state from another test.

Acceptance criteria:

• Identify the smallest reproducible isolation/lifecycle failure, or explicitly document what remains unproven about the original remote collision.
• Reuse the established isolated fixture where appropriate instead of maintaining a separate unscoped memory lifecycle.
• Add deterministic coverage proving per-test database/results isolation and cleanup of central/singleton state; do not rely on repeated probabilistic CI runs as the regression.
• Verify the affected tests with the default-extras Python 3.11 configuration and relevant four-worker selection/full unit command.
• Do not add blanket retries, suppress schema-creation errors, weaken assertions, or change production migration behavior without separate evidence.

Actual Results

ERROR at setup of TestMossBenchDataset.test_dataset_name
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError)
table "PromptMemoryEntries" already exists

The stack goes from setup_memory line 57 (SQLiteMemory()) through schema migration to initial-schema create_all(..., checkfirst=False). The failed job reported 18,481 passed, 125 skipped, 1 error and exit code 2. It ran September 21, 2026, 23:44:33-23:48:28 UTC. The earlier same-SHA/matrix merge-queue job reported 18,482 passed, 125 skipped.

Screenshots

N/A. The linked job contains the setup traceback and test summary.

Versions

• OS/runner: Ubuntu, ubuntu-latest.
• Python: 3.11.
• PyRIT: main SHA 75be23c7b1c9a13f478a60c782d5e4ca63bff885, default dependency configuration (dev matrix cell, not dev_all).
• Browser: N/A.
• Exact dependency versions: use the linked run's installation log and lockfile at that SHA; a pyrit.show_versions() snapshot was not captured.

Related reporting repair: #2776 covers missing SDK JUnit publication. It is a separate task, not proof of the MOSSBench failure mechanism or a prerequisite for an isolated local reproduction.