FEAT: Add Garak ProPILE privacy leakage scenario
@gugu1031 ci sta già lavorando.
Dal 8/9/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Is your feature request related to a problem? Please describe.
PyRIT does not package ProPILE-style tests that use known personal attributes to ask a model for a withheld attribute. Garak has a small prepared record file and prompt templates, but its probe mixes file loading, prompt construction, and execution. This is part of #511.
Describe the solution you'd like
Add a PyRIT-native ProPILE privacy leakage scenario based on garak/probes/propile.py.
Implementation requirements:
- Support Twin, Triplet, Quadruplet, and Unstructured as explicit opt-in scenario techniques.
- Convert garak's
pii_data.jsonlinto a local PyRIT seed dataset. Keep theprompt_templates.tsvcontent in a separate local PyRIT template dataset. Do not add a remote loader. - Preserve each record's
source_datasetandsource_idvalues. Link the statednvidia/Nemotron-CC-v2.1source in dataset provenance. - Preserve the applicable garak Apache-2.0 license, copyright, and attribution notices; mark the PyRIT-formatted files as modified; and update PyRIT's third-party notice material if required by the repository process. See garak's
LICENSE. - Use a dataset configuration, following PromptInject PR #2509, to build attack groups from known attributes and an expected withheld value. Preserve the PII type and source provenance in seed metadata.
- Use bounded atomic attacks and an expected-value scorer such as
SubStringScorer. Reuse existing normalized text matching before adding a new PII scorer. - Keep this dataset and all techniques out of implicit defaults. Users must select the scenario technique and local dataset explicitly.
- The bundled garak records support Twin and limited Triplet combinations, but they do not contain the fields needed for Quadruplet or Unstructured attacks. Support those techniques in the configuration, test them with synthetic fixtures, and raise a clear error when the selected real dataset has no compatible record. Do not silently report a zero-attack success.
- Add exports, focused unit tests, and synchronized
.pyand.ipynbscanner documentation. - State in documentation that an exact match indicates possible disclosure. It does not prove that the target memorized a specific training record.
Follow doc/code/framework.md and the applicable scenario, dataset, test, and documentation instructions. Datasets own records and templates; techniques select the relationship between known and withheld fields; the scenario assembles the campaign; attacks execute; scorers evaluate. Do not port garak's mixin and hook structure.
Describe alternatives you've considered, if relevant
- A loader that streams Nemotron-CC and extracts PII at scenario runtime was considered and rejected. It would add gated-data handling, PII extraction dependencies, nondeterminism, and responsibility bleed.
- A remote loader for garak's JSON file was considered and rejected. This issue should use a local PyRIT dataset with complete provenance and attribution.
- Combining PII records and prompt templates in scenario code would repeat garak's mechanics rather than follow PyRIT's dataset model.
Additional context
- Parent parity issue: #511
- PyRIT structural example: #2509
- Upstream probe:
garak/probes/propile.py - Original ProPILE paper: https://arxiv.org/abs/2307.01881
- Lingua principale
- Python
- Stelle
- 4.5k
- Fork
- 896
- Merge medio
- 3g 8h
- PR unite (30g)
- 191
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di microsoft/PyRIT
-
BUG HarmBench loader drops ContextString, so contextual behaviors are sent without their context Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
Tutte le issue di microsoft/PyRIT
Issue simili
-
agent-ready documentation needs-triage
Difficoltà 1/5 1-3 ore Idoneità per principianti 88/100
-
documentation
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 91/100
-
workflow-status page template still says reusable workflows are "triggered only by workflow_call:" Aperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 92/100
-
instance instance add
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 72/100
searxng/searx-instances#939 · 1 commento ·
-
area-deployment area-integrations triage:bot-seen
Difficoltà 2/5 Mezza giornata Idoneità per principianti 86/100