scan: when the AI reasoning layer fails (e.g. no API key), the report still says "Agentic investigation complete." and lists ai-reasoning as a tool used

Aperta
#26 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
56/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
cli

Direzione di ricerca

Start with scan/reasoner.py, scan/agent_scanner.py, and scan/llm_provider.py, tracing how a failed reasoning run affects the summary and tools_used. Reproduce with no provider key using the documented switch-trust scan command and inspect agentic_trace. Done when the report clearly identifies failed or skipped AI reasoning and does not list an uncompleted layer as a tool.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Note: Setup, reproduction and debugging were done with Claude Code (Anthropic's AI coding assistant), including the root-cause analysis and the suggested fix. I reviewed the findings before filing.

Summary

When the AI reasoning layer fails (e.g. no API key configured), switch-trust scan still reports it as having run: the summary says "Agentic investigation complete.", "Tools" lists ai-reasoning:gemini-3.6-flash, the exit code is 0, and nothing on the console says the AI layer didn't run. A user would reasonably conclude the scan included AI review when it was static-only.

The JSON's agentic_trace does record the failure correctly (exit_reason: "error", error_reason: "auth", 0 tokens), so the information is there — it just isn't surfaced.

Not platform-specific (hit on Windows with the #25 workarounds applied, but nothing in this code path is OS-dependent).

Environment
  • switch-trust-cli 1.7.2 (confirmed in the PyPI wheel and on main at 4e3cac5)
  • ~/.switch-trust/.env with no GENERATOR_MODEL / SCANNER_MODEL / provider key (so the default gemini-3.6-flash)
Steps to reproduce
  1. No LLM provider or API key configured.
  2. switch-trust scan examples/bookstore_agent --output out.json
Expected

Either:

  • the AI layer is skipped with a visible warning. llm_provider.api_key_present() is documented for exactly this ("skip the LLM stage with a warning — the customer never configured one"), and the summary reads like the existing default "AI reasoning skipped (no provider configured)."; or
  • if it's attempted and fails, the summary says so (e.g. "AI reasoning failed (API key missing or invalid) — static findings only.") and ai-reasoning is not listed under Tools.
Actual

Console:

Findings    3
Tools       bandit, opengrep, detect-secrets, ai-reasoning:gemini-3.6-flash
AI summary  Agentic investigation complete.

Exit code 0. JSON scan_metadata:

"ai_summary": "Agentic investigation complete.",
"tools_used": ["bandit", "opengrep", "detect-secrets", "ai-reasoning:gemini-3.6-flash"],
"agentic_trace": {"exit_reason": "error", "error_domain": "user", "error_reason": "auth", "tokens_consumed": 0, ...}

Log: ERROR switch_trust.scan.reasoner: ADK execution failed (ValueError): No API key was provided.

Root cause
  1. Summary ignores exit_reason. scan/reasoner.py: the except block sets exit_reason = "error" and classifies the failure, but line 561 then unconditionally sets summary = "Agentic investigation complete.", and with no final_text that's what is returned.
  2. "Tools used" means "model object built", not "layer ran". scan/agent_scanner.py:789-790 adds ai-reasoning:{model} when llm_model is not None. For the Google provider, make_model() returns the model name string without checking for a key, so it is never None.
  3. The missing-key preflight isn't wired in. scan/llm_provider.py defines api_key_present() (line 84) with a docstring describing the skip-with-warning behaviour, but nothing on the scan path calls it.
  4. The trace comment in reasoner.py says the "stage summary reads [error_domain/error_reason] back to attribute the failure" (_agent_scan_summary), but there is no _agent_scan_summary in the CLI package, so nothing does.
Suggested fix
  • In run_core, call api_key_present(model_string) before make_model(); if it returns False, skip the AI and triage layers, keep the existing "AI reasoning skipped (no provider configured)." summary, and print a visible warning.
  • In reasoner.py, derive summary from exit_reason / error_reason when it isn't "completed" (e.g. "AI reasoning failed (auth) — static findings only.").
  • In agent_scanner.py, add ai-reasoning: / triage: to tools_used only when that layer actually completed (agentic_trace["exit_reason"] == "completed").
  • Optional: a distinct exit code or a --strict flag so CI can tell "static only" from "full scan".

Happy to test a fix.

Lingua principale
Python
Stelle
75
Fork
7
Merge medio
38m
PR unite (30g)
2

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di sandbox-quantum/switch-trust-cli

Tutte le issue di sandbox-quantum/switch-trust-cli

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.