Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Follow-up to #5330: recommended guardrail pattern verified on 0.23.1 (results + one output_info papercut)

Closed
  • Is something wrong?
#5,351 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai-infra-agents, tooling

Research direction

Read the ToolGuardrailFunctionOutput.raise_exception() implementation in agents/tool_guardrails.py and compare it with reject_content(). Reproduce the reported behavior by passing output_info to both methods and inspecting their behavior values; done means the hard-stop path either preserves the reason or warns that the argument is unused. The issue does not name a test file, so locate the relevant guardrail tests before adding coverage.

Written by the indexing model from the issue text.

Description

cc @jbeckwith-oai — following up from #5330. The results were also posted as a comment on the closed thread for the record; reposting here since a closed thread is easy to miss. You suggested the shared-context pattern and the serial configuration — we implemented both on 0.23.1 and verified everything. Results below, plus one small papercut.

  1. Shared state via Runner.run(..., context=state) — confirmed. The plan, halt flag, and observed-call log persist across tool calls and handoffs via data.context.context; no globals needed.

  2. Hard-stop / soft-reject semantics — confirmed, both finer points:

  • Out-of-plan call → raise_exception() → run stops with ToolInputGuardrailTripwireTriggered.
  • Soft-reject: the out-of-plan call sets the halt flag and returns reject_content(); the run completes and the model reports which calls succeeded and which were blocked. Later calls (even in-plan ones) are rejected by the halt-flag check while the model wraps up.
  • Fresh-object-per-run isolation confirmed: a second run with a fresh state object runs clean after a first run halted; reusing the halted state object keeps blocking.
  1. The race window, quantified. We injected 6 tool calls (4 in-plan, 2 out-of-plan) directly into the batch executor Runner.run uses internally — no LLM, n=30 per configuration:
  • max_function_tool_concurrency=1: execution order matched submission order, zero timeline overlap in 30/30 runs; at tripwire time only 3/6 calls had started — the window cannot open under serial execution.
  • Default concurrent setting: timeline overlap in 30/30 runs; all 6 calls had already started at tripwire time — the window does reproduce.
  • Worst case: 5 high-risk out-of-plan calls (money transfer, file deletion, log purge, bank wire, DB drop) → 5/5 vetoed with zero side-effect leakage in both serial and concurrent settings (n=30 each).

  1. One papercut for anyone implementing this: a hard-stop tripwire can't carry its reason —

    from agents.tool_guardrails import ToolGuardrailFunctionOutput

    out = ToolGuardrailFunctionOutput.raise_exception(output_info={"rule": "R1_plan_breach"})
    print(out.behavior) # {'type': 'raise_exception'} — output_info dropped

    ok = ToolGuardrailFunctionOutput.reject_content(message="blocked", output_info={"rule": "R1"})
    print(ok.behavior) # {'type': 'reject_content', 'message': 'blocked'} — message kept

We pass the veto reason through the shared context object instead; either surfacing output_info in the behavior, or warning on the unused kwarg, would fix it.

  1. Coverage boundary (documented explicitly on our side): tool input guardrails don't intercept hosted tools or handoffs — coverage claims are scoped to local function tools.

Environment: openai-agents 0.23.1 (latest release), Python 3.12.13, Debian 13. Repro scripts and full result artifacts: https://github.com/lsy1533133/phm-agent-verifier-tests

Dominant language
Python
Stars
30k
Forks
4.9k
Avg merge
16h 25m
Merged PRs (30d)
220

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/openai-agents-python

All issues in openai/openai-agents-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.