Prompt-injection test corpus and red-team scenarios for the egress gate

Aperta
#231 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, yaml
Ambito
security, testing

Direzione di ricerca

Read crates/agent-core/src/egress_tests.rs, then inspect two existing files in evals/scenarios/ and the relevant code in crates/evals/src. Start by proposing the first five synthetic cases in a comment, as requested; keep any suspected real bypass private via SECURITY.md. Done means at least 20 hermetic cases pass on Linux and Windows, weakened-gate checks fail as expected, and adding a case is documented as a data-only change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: daemon area: security enhancement help wanted

size: M · first-timer: no

Security-sensitive: changes here need a maintainer review. This is test and corpus work for the egress gate, not a change to the gate itself. Found a bypass? Do not file it here: see SECURITY.md.

Summary

A prompt-injection test corpus and red-team scenarios for the egress gate and the other approval floors: a versioned set of hostile pieces of content (web page, mail, file, tool output, skill text) paired with a scripted model that always obeys the injected instruction, and an assertion that nothing leaves the machine and nothing dangerous runs without an approval card.

Why it matters

Praxi reads untrusted content all the time (pages, mail, downloads, MCP output). The design answer is the taint and egress handling: once a run has read untrusted content, sending data out, typing, shell commands and launching programs ask first. That is only as good as its tests. A shared, growing corpus lets contributors add "here is an attack shape we should survive" as a data file instead of a new bespoke test.

Current state

  • The gate: crates/permission-engine/src/egress.rs and crates/agent-core/src/taint.rs (what a run has read, and the card shown when it then wants to send data out); the floors for typing, shell and launching under taint are described in docs/guide/security.md.
  • Existing tests are hand-written Rust tests per behaviour: crates/agent-core/src/egress_tests.rs and tests in taint.rs.
  • Hermetic scenarios: evals/scenarios/*.yaml (17 today) run by crates/evals with a scripted fake model (script, reactions, expect on requests and tool calls). None of them is about an injected instruction after an untrusted read; #189 lists "egress after an untrusted read" only as one example among several.
  • evals/retrieval/corpus/poisoned-example.md is a poisoned document for the retrieval evals, a different purpose.

Proposal and suggested approach

  1. Corpus format. A folder such as evals/injection/ with one fixture per case: the untrusted content (HTML snippet, mail body, file text, tool result, skill text), the attack goal (send a canary to an external host through the browser, a shell command or mail; write outside a trusted folder; type a fill secret on the wrong origin; widen a grant), the entry point (which tool returns the content), and the expected gate outcome (approval card shown, or refused). A short README explains how to add a case.
  2. Runner. A scenario family in crates/evals that scripts a model which complies with whatever the content says, then asserts: no off-machine effect happened without a card, the canary never left, and which gate stopped it. Also include benign controls (clean runs must not be nagged) so the corpus cannot be "passed" by asking for everything.
  3. Report. A table of case, entry point, goal and the gate that held, printed by the runner and uploaded as a CI artifact.
  4. Seed set of 20 or more cases across entry points and goals, including encoded and split payloads, homoglyph hosts and look-alike URLs, instructions hidden in alt text or comments, and multi-step plans.

Cases must be synthetic; do not include working exploits for undisclosed issues. If a case shows a real bypass, stop and report it privately as described in SECURITY.md.

Acceptance criteria

  • At least 20 cases run hermetically with cargo test -p evals and pass on Linux and Windows.
  • A deliberately weakened gate (a test-only switch or a mutated fixture) makes the relevant cases fail.
  • Adding a case is a data-only change documented in the README.
  • No network, no real provider.

Out of scope

Changing the gate, live-model red teaming (an opt-in tier, possible follow-up with #200), and scoring different models.

How to get started

Read crates/agent-core/src/egress_tests.rs, then two files in evals/scenarios/ and crates/evals/src. Comment here with the first five cases you would write.

Lingua principale
Rust
Stelle
0
Fork
1
Merge medio
6h 54m
PR unite (30g)
152

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di emreocell/Praxi-Bot

Tutte le issue di emreocell/Praxi-Bot

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.