scan: when the AI reasoning layer fails (e.g. no API key), the report still says "Agentic investigation complete." and lists ai-reasoning as a tool used
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 56/100
Direzione di ricerca
Start with scan/reasoner.py, scan/agent_scanner.py, and scan/llm_provider.py, tracing how a failed reasoning run affects the summary and tools_used. Reproduce with no provider key using the documented switch-trust scan command and inspect agentic_trace. Done when the report clearly identifies failed or skipped AI reasoning and does not list an uncompleted layer as a tool.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Note: Setup, reproduction and debugging were done with Claude Code (Anthropic's AI coding assistant), including the root-cause analysis and the suggested fix. I reviewed the findings before filing.
Summary
When the AI reasoning layer fails (e.g. no API key configured), switch-trust scan still reports it as having run: the summary says "Agentic investigation complete.", "Tools" lists ai-reasoning:gemini-3.6-flash, the exit code is 0, and nothing on the console says the AI layer didn't run. A user would reasonably conclude the scan included AI review when it was static-only.
The JSON's agentic_trace does record the failure correctly (exit_reason: "error", error_reason: "auth", 0 tokens), so the information is there — it just isn't surfaced.
Not platform-specific (hit on Windows with the #25 workarounds applied, but nothing in this code path is OS-dependent).
Environment
switch-trust-cli1.7.2 (confirmed in the PyPI wheel and onmainat4e3cac5)~/.switch-trust/.envwith noGENERATOR_MODEL/SCANNER_MODEL/ provider key (so the defaultgemini-3.6-flash)
Steps to reproduce
- No LLM provider or API key configured.
switch-trust scan examples/bookstore_agent --output out.json
Expected
Either:
- the AI layer is skipped with a visible warning.
llm_provider.api_key_present()is documented for exactly this ("skip the LLM stage with a warning — the customer never configured one"), and the summary reads like the existing default"AI reasoning skipped (no provider configured)."; or - if it's attempted and fails, the summary says so (e.g.
"AI reasoning failed (API key missing or invalid) — static findings only.") andai-reasoningis not listed under Tools.
Actual
Console:
Findings 3
Tools bandit, opengrep, detect-secrets, ai-reasoning:gemini-3.6-flash
AI summary Agentic investigation complete.
Exit code 0. JSON scan_metadata:
"ai_summary": "Agentic investigation complete.",
"tools_used": ["bandit", "opengrep", "detect-secrets", "ai-reasoning:gemini-3.6-flash"],
"agentic_trace": {"exit_reason": "error", "error_domain": "user", "error_reason": "auth", "tokens_consumed": 0, ...}
Log: ERROR switch_trust.scan.reasoner: ADK execution failed (ValueError): No API key was provided.
Root cause
- Summary ignores
exit_reason.scan/reasoner.py: theexceptblock setsexit_reason = "error"and classifies the failure, but line 561 then unconditionally setssummary = "Agentic investigation complete.", and with nofinal_textthat's what is returned. - "Tools used" means "model object built", not "layer ran".
scan/agent_scanner.py:789-790addsai-reasoning:{model}whenllm_model is not None. For the Google provider,make_model()returns the model name string without checking for a key, so it is neverNone. - The missing-key preflight isn't wired in.
scan/llm_provider.pydefinesapi_key_present()(line 84) with a docstring describing the skip-with-warning behaviour, but nothing on the scan path calls it. - The trace comment in
reasoner.pysays the "stage summary reads [error_domain/error_reason] back to attribute the failure" (_agent_scan_summary), but there is no_agent_scan_summaryin the CLI package, so nothing does.
Suggested fix
- In
run_core, callapi_key_present(model_string)beforemake_model(); if it returnsFalse, skip the AI and triage layers, keep the existing"AI reasoning skipped (no provider configured)."summary, and print a visible warning. - In
reasoner.py, derivesummaryfromexit_reason/error_reasonwhen it isn't"completed"(e.g."AI reasoning failed (auth) — static findings only."). - In
agent_scanner.py, addai-reasoning:/triage:totools_usedonly when that layer actually completed (agentic_trace["exit_reason"] == "completed"). - Optional: a distinct exit code or a
--strictflag so CI can tell "static only" from "full scan".
Happy to test a fix.
- Lingua principale
- Python
- Stelle
- 75
- Fork
- 7
- Merge medio
- 38m
- PR unite (30g)
- 2
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di sandbox-quantum/switch-trust-cli
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
sandbox-quantum/switch-trust-cli#25 · 2 commenti ·
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
sandbox-quantum/switch-trust-cli#6 · 2 commenti ·
Tutte le issue di sandbox-quantum/switch-trust-cli
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
mohanraj00/entelos#200 ·
I maintainer di solito rispondono entro 1 giorno
-
Dead url in mise.tomlAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 68/100
-
sweep
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
bketelsen/truenas-mcp#77 ·
I maintainer di solito rispondono entro 1 giorno
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 86/100
sandbox-quantum/switch#2163 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
openedx/xblock-lti-consumer#765 ·
I maintainer di solito rispondono entro 1 giorno