buzz-acp: idle-pool teardown race silently swallows a concurrently arriving message (no log trace beyond routine steer ack)

Open
#7,720 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust

Research direction

Read idle_pool_sleep_due in crates/buzz-acp/src/lib.rs:2164-2179, then trace the reaper select! arm at lib.rs:3606-3654 through shutdown_agent_pool and the SteerAck branches at lib.rs:3889-3922. Reproduce a message arriving as the 900-second teardown begins; done means it is deferred, dispatched to the fresh pool, re-queued, or surfaced as an error rather than silently dropped.

Written by the indexing model from the issue text.

Description

Summary

buzz-acp's idle-pool reaper (idle_pool_sleep_due, crates/buzz-acp/src/lib.rs:2164-2179, invoked from the reaper-tick select! arm at lib.rs:3606-3654) tears the agent pool back to a lazy/listening state after idle_pool_sleep_secs (900s in our config) of inactivity via shutdown_agent_pool. When a new message/wake arrives at, or just before, the same tick that triggers this teardown, the message can be silently dropped: no error, no retry record, nothing but a routine tracing::info! line —

crates/buzz-acp/src/lib.rs:3914-3922
tracing::info!(
    channel = %channel_id,
    event_id = %event_id,
    ?ack,
    release_withheld,
    drop_withheld,
    signal_fallback,
    "non-cancelling steer ack received"
);

— which reads as routine steer bookkeeping, not as a dropped-message signal.

This looks like a distinct failure mode from the already-tracked #6378 / #7171. Those describe last_activity not being refreshed for genuinely in-flight work (background subagents, long turns), so the pool is torn down while the agent is still busy. Here, by contrast, the pool really is idle and due for teardown — the bug is that a message arriving concurrently with that teardown races shutdown_agent_pool / pool rebuild and is lost instead of being deferred, dispatched to the fresh pool, or re-queued.

Scope

Affects every buzz-acp managed agent, independent of subscribe mode (Mentions vs Config). Mentions-mode agents hit it far more often in practice per-wake — with fewer incidental messages to keep the pool warm between targeted wakes, the pool has usually gone fully idle and been torn down by the time the next @mention/wake lands, so the race window is entered more often relative to total wake attempts.

Evidence

The "non-cancelling steer ack received" log line is the direct fingerprint of a steer racing pool teardown/rebuild. Counts observed in our deployment's agent logs:

  • harness-dev-orch: 69 occurrences (highest raw count, but least visible impact — many subscribed channels mean other traffic quickly re-wakes the pool, masking the drop)
  • stock_orch: 8 occurrences
  • stock-news: 5 occurrences

Real-world impact

stock-news's pre-market briefing pipeline depends on a cron-triggered T-30 wake message. On 2026-09-16, 09-17, and 09-18, this wake was accepted by the relay (accepted:true) but the agent never responded to it — 3 relay retries, then retry-exhausted — consistent with the wake landing inside an idle-pool teardown window and being swallowed.

Suggested repro / investigation

Dispatch a message exactly at/after idle_pool_sleep_bound elapses (900s default) while pool_ready is true and the idle_pool_sleep_due gates (work_queued, prompt_tasks_in_flight, wake_or_respawn_in_flight) are all clear, and trace what happens to that message through shutdown_agent_pool and pool rebuild (lib.rs:3635-3653). The race window looks like it's between the gate check passing and the teardown/rebuild actually completing, rather than in the gate logic itself — i.e. a message dispatched into the old pool concurrently with (or immediately after) shutdown_agent_pool gets a steer ack against a pool that's mid-teardown, hits one of the non-fallback SteerAck branches (lib.rs:3889-3913), and is neither retried against the new pool nor surfaced as an error.

Related issues

  • #6378 — idle pool teardown kills an in-flight session, losing a queued background-task wakeup. Related mechanism (same reaper), different trigger: in-flight session activity not counted as activity, vs. a new message racing the teardown of an already-idle pool.
  • #7171 — idle-pool reaper tears down right after a long turn completes. Same reaper code path, different root cause: stale last_activity not refreshed on turn completion, vs. a delivery race during an already-correctly-triggered teardown.

Neither of the above describes this specific scenario — a message that arrives concurrently with an already-due idle teardown being silently swallowed with no log trace beyond a routine steer-ack info line — so filing separately rather than piling onto either.

Dominant language
Rust
Stars
33.7k
Forks
4.4k
Avg merge
1d 21h
Merged PRs (30d)
239

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from block/buzz

All issues in block/buzz

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.