Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Design] Risk triage for tool calls: what does it classify, and where does it sit in the permission engine?

Aperta
  • Qualcosa non va?
#245 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@rifkir23 ci sta già lavorando.

Dal 10/10/2026.

  • #257 di @rifkir23 — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
python
Ambito
ai-infra-agents, backend-api-design

Direzione di ricerca

Read core/harness/permissions.py and the permission flow in core/agent_runtime/runner.py, starting with the precedence and fail-closed behavior described in the issue. Decide whether risk means content triage, action triage, or neither, and settle the integration point and failure behavior with the maintainers. The issue’s linked open pull request #257 indicates work is already underway.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

For context: #189 (feat(classifier): harness risk classifier (split from #181)) was closed with "Auto-mode-style risk triage is a real product question, but it needs a design discussion first (framing of the classifier prompt, where it sits in the permission engine, integration tests). Happy to have that conversation in an issue before any code." This is that issue — no code attached on purpose.

1. Why the prototype answered none of those three questions

#189 added core/harness/classifier.py: a rule-based (keyword + boundary heuristic) scorer with classify(text) -> RiskDecision, gated behind DEEPCODE_RISK_CLASSIFIER, ~427 lines, and no caller. So "where does it sit in the permission engine" was not answerable from the diff — it was a library with no integration point. That is the first thing to fix in the design, before any refactor.

2. The ambiguity worth settling first: two things are called "risk"

(a) Untrusted-content triage. Input is text (tool output, MCP payload, injected input); the question is "should this content be trusted as instructions?" It is consumed before the model acts on it. This is not a permission decision — the permission engine never sees content, only a tool call. Its natural home is a note/reminder channel or a pre-model transform.

(b) Action triage / auto-mode gating. Input is a tool call (name + canonicalized arguments); the question is "may this run without asking?" This is exactly what core/harness/permissions.py already decides (allow / ask / deny).

#189 built (a). The review read it as (b). Both are defensible products, but they have different seams, different inputs, and different tests, so the first decision is which one is wanted — and (b) is the one that has a natural home in the engine.

3. If (b): where it sits, and what it must never break

Current precedence in core/harness/permissions.py (first decisive wins):

  1. sensitive-path protection → DENY
  2. canonical Read-only upper bound → DENY
  3. explicit rule match (last-match-wins) → its action
  4. mode default for read-only vs mutating tools

Three placements, with the invariant each one buys:

  • Option A (my preference): a strictness-raising evaluator after step 4. It may turn allow → ask. It may never turn deny → allow, and never reinterpret steps 1–2. Invariant: a classifier can only make the engine more careful, never less — so a user's explicit rule and the sensitive-path boundary stay untouched, and full_auto ("rules only, no implicit ask") becomes "rules, plus a soft ask when the classifier objects" rather than losing its promise.
  • Option B: a rule family inside step 3, participating in last-match-wins. More expressive, but then a classifier decision can silently outrank a user's explicit allow, which I'd argue against.
  • Option C: not in the permission engine at all — advisory only, along the repeat_guard / no-progress-reminder channel (a context_note, no gate). Cheapest, adds zero new failure modes, and protects nothing in full_auto; it just tells the model.

Whatever is chosen has to preserve two existing behaviors: the runner's fail-closed path (permission_checker raising ⇒ "permission check errored (fail-closed)", core/agent_runtime/runner.py:1501), and headless ask ⇒ deny ("autonomy never silently escalates", runner.py:1497).

4. If a model is involved: prompt framing

  • Input: tool name + canonicalized arguments. Deliberately not tool output — that would add a second-order injection surface and an unbounded input to the permission path (this is the one place where (a) and (b) genuinely interact).
  • Output: strict schema ({"risk": "low"|"high", "reason": "<one short line>"}), no free-form prose, no tool access, no ability to emit a permission decision directly — it returns an opinion, the engine decides.
  • Failure mode: timeout / unparsable / provider error. Default-off feature ⇒ I'd fail-closed (ask) only when explicitly enabled, and keep the existing "error ⇒ deny" path for genuinely raised exceptions. Worth agreeing explicitly.
  • Cost/latency: a model call inside the permission path is per-tool-call latency. It needs caching by call signature and a strict timeout, otherwise it is a UX regression in exactly the mode it targets.
  • Rules vs model: rule-based scoring (what #189 did) is deterministic and free but has a high false-negative rate on anything but the patterns it was written for. "Rules only", "model only", or "rules first, model only on the residue" are three different products here — if it is rules-only, most of this section disappears.

5. Integration tests I would write

  1. The classifier cannot loosen anything: sensitive-path DENY and plan-mode DENY survive an allow opinion (this is the regression test for Option A's invariant).
  2. The intended path: full_auto + enabled + high risk ⇒ ask; with no approval callback (headless) ⇒ denied with a reason.
  3. Classifier failure is contained: raising, timing out, or returning garbage ⇒ fail-closed, never a crash, never an allow.
  4. No waste: no classifier call when steps 1–3 already decided; at most one call per call signature (cache hit asserted).
  5. Evasion fixtures (encoded/obfuscated arguments, benign wrappers) — plus an explicit statement that this is advisory, not a security boundary.

6. What I need from you

  1. (a) content triage, (b) action triage, or neither?
  2. If (b): A, B, or C — and default-off?
  3. If model-based: is a second LLM call per tool call acceptable in full_auto, or must it stay rules-only?
  4. Anything about upstream's roadmap that already covers this (searching for keyring/repeat_guard/auto-mode/risk found no open PR or issue on it — #181 is closed).

#189's prototype can be rebased onto whatever shape comes out of this discussion; I am not opening a code PR before that.

Lingua principale
Python
Stelle
16.7k
Fork
2.2k
Merge medio
2g 20h
PR unite (30g)
15

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di HKUDS/DeepCode

Tutte le issue di HKUDS/DeepCode

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.