Risk Agent help text promises a "show it" control that no longer exists

Open Beginner friendly
#546 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
Under an hour
Newbie friendliness
76/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Domain
documentation

Research direction

Start in the agent Runtime settings UI and locate the Output Guardrail description under Guardrails. Update the help text to match the current verdict behavior, including that flagged answers are replaced in chat and remain available in the trace. Reproduce the listed steps to verify the rendered description no longer promises a show control.

Written by the indexing model from the issue text.

Description

Describe the bug

The output-guardrail help text in the agent Runtime settings tells the user a flagged answer can be revealed on demand:

"Checks the agent's answer once it is complete. A flagged answer is withheld in chat until you choose to show it, alongside the detected risks."

That control was deliberately removed. The change that shipped the current verdict UI states:

"A flagged answer is now withheld outright: 'Show answer' is gone, so the verdict replaces the answer rather than hiding it behind a toggle. The answer text itself is untouched and still readable in the trace."

So the settings panel describes behaviour the product no longer has. A user configuring an output guard would reasonably expect to be able to inspect a withheld answer in the chat, and there is no way to do so.

Impact. Small but specific: it sets a wrong expectation at exactly the moment someone is deciding whether to turn the guard on. Anyone choosing Risk Agent partly because they believed they could still review flagged answers in place has been misled about what they are enabling. The answer does remain readable in the trace, so the information is not lost — but that is a different surface with different access, and the help text does not say so.

Not affected: the guard itself, which runs and reports correctly.

Steps to reproduce the behavior
1. Open any pydantic_ai agent → Runtime tab → Edit.
2. Scroll to the Guardrails section.
3. Set Output Guardrail to "Risk Agent".
4. Read the description rendered under the dropdown.

Observed: "A flagged answer is withheld in chat until you choose to show it, alongside
the detected risks."

There is no control in the chat view to show a withheld answer.
Expected Behavior

The help text matches the shipped behaviour — a flagged answer is withheld and replaced by the verdict and its risk categories, with the full text available in the trace rather than in the chat.

Relevant log output
Environment
- Env: dev (dev.design.akd.odsi.io)
- Browser: Chrome
- Date observed: 2026-08-25

No errors. This is a text/behaviour mismatch in the settings panel, not a runtime fault.

Found while running the output guardrail end to end: the guard was verified to run
(verdict recorded on every turn, provider "RiskAgent", latency ~5.7s) and to pass content
correctly. Only the description is wrong.
Dominant language
Jupyter Notebook
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NASA-IMPACT/akd-core

All issues in NASA-IMPACT/akd-core

Similar issues

More Documentation issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.