Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

FEAT GUI: Expose configured objective-scorer presets

Aperta
#2,755 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

Start with pyrit/registry/components/scorer_registry.py, pyrit/setup/initializers/scorers.py, and Scenario._get_default_objective_scorer(); trace how configured instances, presets, tags, and result families are represented. Review the existing /scores routes and model/Python/scorer/test instructions, then add fake-registered-scorer tests covering empty or incompatible registries, refusal polarity, credential safety, and the absence of scoring or model calls.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature-request GUI not ready yet
Is your feature request related to a problem? Please describe.

The custom-scenario flow needs a simple scorer choice. PyRIT already has ScorerRegistry and ScorerInitializer presets, but the GUI currently has no read API for choosing configured objective-scorer instances.

Work item 12 of 17 in #2744. This prepares the scenario milestone and remains not ready yet until promoted after item 11.

Describe the solution you'd like

Expose a small typed API for initialized, compatible objective-scorer presets from ScorerRegistry.instances.

  • Return stable registry names, useful descriptions, scorer/result family, relevant tags, compatibility/prerequisite information, and the compatible configured default_objective_scorer selection.
  • Offer ready instances, not just buildable class names. Listing must not instantiate scorers, invoke targets, or manufacture credential-backed configuration.
  • Reuse existing presets/composites where available, such as inverted_refusal, task-achieved variants, acs_with_refusal, and scale_and_refusal. Availability depends on initialized targets and ScorerInitializer.
  • Preserve objective-result meaning. Raw refusal=true means refusal and must not be exposed as attack success; use an appropriate existing inverter/composite or clearly validated objective preset.
  • Report an empty/unusable registry clearly so the GUI can guide setup. Do not silently choose an incompatible default or expose secrets through identifiers/configuration metadata.

Acceptance criteria:

  • The API distinguishes configured instances from merely discoverable classes.
  • Compatible default tags and non-default presets are represented correctly.
  • Missing dependencies, empty registries, incompatible result families, and refusal polarity are covered.
  • The response contains no credential values and listing performs no scoring/model calls.
  • Tests use fake registered scorers and require no paid services.
Describe alternatives you've considered, if relevant

Do not create a hardcoded frontend scorer list, a general scorer builder, a new scorer registry, or an unscored scenario mode. One selected objective scorer may already be a composite; arbitrary multi-scorer aggregation is out of scope.

Additional context

Start with pyrit\registry\components\scorer_registry.py, pyrit\setup\initializers\scorers.py, and Scenario._get_default_objective_scorer(). The existing /scores routes concern score operations, not a configured-scorer catalog. Follow model/Python/scorer/test instructions and keep registry responsibility separate from service DTOs.

Lingua principale
Python
Stelle
4.5k
Fork
896
Merge medio
3g 8h
PR unite (30g)
191

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/PyRIT

Tutte le issue di microsoft/PyRIT

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.