FEAT GUI: Expose configured objective-scorer presets
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 45/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- python
- Ambito
- ai, api, backend-api-design
Direzione di ricerca
Start with pyrit/registry/components/scorer_registry.py, pyrit/setup/initializers/scorers.py, and Scenario._get_default_objective_scorer(); trace how configured instances, presets, tags, and result families are represented. Review the existing /scores routes and model/Python/scorer/test instructions, then add fake-registered-scorer tests covering empty or incompatible registries, refusal polarity, credential safety, and the absence of scoring or model calls.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Is your feature request related to a problem? Please describe.
The custom-scenario flow needs a simple scorer choice. PyRIT already has ScorerRegistry and ScorerInitializer presets, but the GUI currently has no read API for choosing configured objective-scorer instances.
Work item 12 of 17 in #2744. This prepares the scenario milestone and remains not ready yet until promoted after item 11.
Describe the solution you'd like
Expose a small typed API for initialized, compatible objective-scorer presets from ScorerRegistry.instances.
- Return stable registry names, useful descriptions, scorer/result family, relevant tags, compatibility/prerequisite information, and the compatible configured default_objective_scorer selection.
- Offer ready instances, not just buildable class names. Listing must not instantiate scorers, invoke targets, or manufacture credential-backed configuration.
- Reuse existing presets/composites where available, such as inverted_refusal, task-achieved variants, acs_with_refusal, and scale_and_refusal. Availability depends on initialized targets and ScorerInitializer.
- Preserve objective-result meaning. Raw refusal=true means refusal and must not be exposed as attack success; use an appropriate existing inverter/composite or clearly validated objective preset.
- Report an empty/unusable registry clearly so the GUI can guide setup. Do not silently choose an incompatible default or expose secrets through identifiers/configuration metadata.
Acceptance criteria:
- The API distinguishes configured instances from merely discoverable classes.
- Compatible default tags and non-default presets are represented correctly.
- Missing dependencies, empty registries, incompatible result families, and refusal polarity are covered.
- The response contains no credential values and listing performs no scoring/model calls.
- Tests use fake registered scorers and require no paid services.
Describe alternatives you've considered, if relevant
Do not create a hardcoded frontend scorer list, a general scorer builder, a new scorer registry, or an unscored scenario mode. One selected objective scorer may already be a composite; arbitrary multi-scorer aggregation is out of scope.
Additional context
Start with pyrit\registry\components\scorer_registry.py, pyrit\setup\initializers\scorers.py, and Scenario._get_default_objective_scorer(). The existing /scores routes concern score operations, not a configured-scorer catalog. Follow model/Python/scorer/test instructions and keep registry responsibility separate from service DTOs.
- Lingua principale
- Python
- Stelle
- 4.5k
- Fork
- 896
- Merge medio
- 3g 8h
- PR unite (30g)
- 191
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di microsoft/PyRIT
-
BUG HarmBench loader drops ContextString, so contextual behaviors are sent without their context Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
Tutte le issue di microsoft/PyRIT
Issue simili
-
agent-ready documentation needs-triage
Difficoltà 1/5 1-3 ore Idoneità per principianti 88/100
-
documentation
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 91/100
-
workflow-status page template still says reusable workflows are "triggered only by workflow_call:" Aperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 92/100
-
instance instance add
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 72/100
searxng/searx-instances#939 · 1 commento ·
-
area-deployment area-integrations triage:bot-seen
Difficoltà 2/5 Mezza giornata Idoneità per principianti 86/100