Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Design] Risk triage for tool calls: what does it classify, and where does it sit in the permission engine?

Đang mở
  • Có gì không ổn?
#245 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 3 ngày

@rifkir23 đang làm issue này rồi.

Từ ngày 10/10/2026.

  • #257 của @rifkir23 — đang mở

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
20/100
Loại issue
Tính năng
Độ rõ ràng
Cần làm rõ
Mức độ hoạt động
Đình trệ
Công nghệ
python
Lĩnh vực
ai-infra-agents, backend-api-design

Hướng nghiên cứu

Read core/harness/permissions.py and the permission flow in core/agent_runtime/runner.py, starting with the precedence and fail-closed behavior described in the issue. Decide whether risk means content triage, action triage, or neither, and settle the integration point and failure behavior with the maintainers. The issue’s linked open pull request #257 indicates work is already underway.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

For context: #189 (feat(classifier): harness risk classifier (split from #181)) was closed with "Auto-mode-style risk triage is a real product question, but it needs a design discussion first (framing of the classifier prompt, where it sits in the permission engine, integration tests). Happy to have that conversation in an issue before any code." This is that issue — no code attached on purpose.

1. Why the prototype answered none of those three questions

#189 added core/harness/classifier.py: a rule-based (keyword + boundary heuristic) scorer with classify(text) -> RiskDecision, gated behind DEEPCODE_RISK_CLASSIFIER, ~427 lines, and no caller. So "where does it sit in the permission engine" was not answerable from the diff — it was a library with no integration point. That is the first thing to fix in the design, before any refactor.

2. The ambiguity worth settling first: two things are called "risk"

(a) Untrusted-content triage. Input is text (tool output, MCP payload, injected input); the question is "should this content be trusted as instructions?" It is consumed before the model acts on it. This is not a permission decision — the permission engine never sees content, only a tool call. Its natural home is a note/reminder channel or a pre-model transform.

(b) Action triage / auto-mode gating. Input is a tool call (name + canonicalized arguments); the question is "may this run without asking?" This is exactly what core/harness/permissions.py already decides (allow / ask / deny).

#189 built (a). The review read it as (b). Both are defensible products, but they have different seams, different inputs, and different tests, so the first decision is which one is wanted — and (b) is the one that has a natural home in the engine.

3. If (b): where it sits, and what it must never break

Current precedence in core/harness/permissions.py (first decisive wins):

  1. sensitive-path protection → DENY
  2. canonical Read-only upper bound → DENY
  3. explicit rule match (last-match-wins) → its action
  4. mode default for read-only vs mutating tools

Three placements, with the invariant each one buys:

  • Option A (my preference): a strictness-raising evaluator after step 4. It may turn allow → ask. It may never turn deny → allow, and never reinterpret steps 1–2. Invariant: a classifier can only make the engine more careful, never less — so a user's explicit rule and the sensitive-path boundary stay untouched, and full_auto ("rules only, no implicit ask") becomes "rules, plus a soft ask when the classifier objects" rather than losing its promise.
  • Option B: a rule family inside step 3, participating in last-match-wins. More expressive, but then a classifier decision can silently outrank a user's explicit allow, which I'd argue against.
  • Option C: not in the permission engine at all — advisory only, along the repeat_guard / no-progress-reminder channel (a context_note, no gate). Cheapest, adds zero new failure modes, and protects nothing in full_auto; it just tells the model.

Whatever is chosen has to preserve two existing behaviors: the runner's fail-closed path (permission_checker raising ⇒ "permission check errored (fail-closed)", core/agent_runtime/runner.py:1501), and headless ask ⇒ deny ("autonomy never silently escalates", runner.py:1497).

4. If a model is involved: prompt framing

  • Input: tool name + canonicalized arguments. Deliberately not tool output — that would add a second-order injection surface and an unbounded input to the permission path (this is the one place where (a) and (b) genuinely interact).
  • Output: strict schema ({"risk": "low"|"high", "reason": "<one short line>"}), no free-form prose, no tool access, no ability to emit a permission decision directly — it returns an opinion, the engine decides.
  • Failure mode: timeout / unparsable / provider error. Default-off feature ⇒ I'd fail-closed (ask) only when explicitly enabled, and keep the existing "error ⇒ deny" path for genuinely raised exceptions. Worth agreeing explicitly.
  • Cost/latency: a model call inside the permission path is per-tool-call latency. It needs caching by call signature and a strict timeout, otherwise it is a UX regression in exactly the mode it targets.
  • Rules vs model: rule-based scoring (what #189 did) is deterministic and free but has a high false-negative rate on anything but the patterns it was written for. "Rules only", "model only", or "rules first, model only on the residue" are three different products here — if it is rules-only, most of this section disappears.

5. Integration tests I would write

  1. The classifier cannot loosen anything: sensitive-path DENY and plan-mode DENY survive an allow opinion (this is the regression test for Option A's invariant).
  2. The intended path: full_auto + enabled + high risk ⇒ ask; with no approval callback (headless) ⇒ denied with a reason.
  3. Classifier failure is contained: raising, timing out, or returning garbage ⇒ fail-closed, never a crash, never an allow.
  4. No waste: no classifier call when steps 1–3 already decided; at most one call per call signature (cache hit asserted).
  5. Evasion fixtures (encoded/obfuscated arguments, benign wrappers) — plus an explicit statement that this is advisory, not a security boundary.

6. What I need from you

  1. (a) content triage, (b) action triage, or neither?
  2. If (b): A, B, or C — and default-off?
  3. If model-based: is a second LLM call per tool call acceptable in full_auto, or must it stay rules-only?
  4. Anything about upstream's roadmap that already covers this (searching for keyring/repeat_guard/auto-mode/risk found no open PR or issue on it — #181 is closed).

#189's prototype can be rebased onto whatever shape comes out of this discussion; I am not opening a code PR before that.

Ngôn ngữ chính
Python
Star
16.7k
Fork
2.2k
Merge trung bình
2 ngày 20 giờ
Pull request đã merge (30 ngày)
15

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của HKUDS/DeepCode

Tất cả issue của HKUDS/DeepCode

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.