Retry layer misclassifies deterministic 400s as transient, workflow steps replay non-idempotent work, and AgentTeam.wait_for_completions() loses finished sub-agents

Đang mở
#5,755 5 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
30/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
backend

Hướng nghiên cứu

This bundles three separate bugs. Start with classify_error_kind in llm/llm.py (around lines 977-1017) and the retry loops _call_with_retry and _call_with_retry_async, then Workflow step retry in workflows/workflows.py (around 3360) and wait_for_completions / _handle_sub_agent_completion in agents/agents.py (around 4461). Done means a 400 or programming error fails on the first attempt, a non-transient function step runs once, and wait_for_completions() returns already-finished sub-agents with a bounded _completion_events. Split it into separate PRs and agree the retry defaults with maintainers first.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug claude documentation origin/outsider

Summary

An analysis of src/praisonai-agents/praisonaiagents found three validated gaps in how failures and completions are handled. They follow the same pattern: failures and completions are inferred from loose signals (substrings, "any exception", deleted records) instead of the real signal. All three were confirmed against main at f8bf0921. Findings 1 and 2 have a reproduction below. Finding 3 is confirmed by code reading and the agent's own reproduction.

Scope: core SDK only. Docs, tests and file size are out of scope.


1. classify_error_kind matches bare digits in messages, so deterministic 400s are retried as "overloaded" or "rate_limit"

Where: llm/llm.py:977-979 (rate_limit) and llm/llm.py:1015-1017 (overloaded)

# rate limit
if any(indicator in error_str for indicator in [
    "rate limit", "ratelimit", "too many request", "resource_exhausted",
    "usage limit", "429"                      # <- matches "6429", "max 4290", ...
]) or "429" in str(getattr(error, "status_code", "")):
    return "rate_limit"
...
# overloaded
"503", "502", "500"                           # <- matches "max_tokens=5000"

status_code is only consulted for 429. A 400 whose message contains "500", "502", "503" or "429" is treated as transient. _call_with_retry and _call_with_retry_async then retry it with backoff (llm/llm.py:~1473 and ~1559).

Reproduction (stubbed litellm, no network):

class E(Exception): status_code = 400
o = LLM.__new__(LLM)
o.classify_error_kind(E("Invalid value: max_tokens=5000 exceeds limit 4096"))  # -> 'overloaded'
o.classify_error_kind(E("user id 6429 not permitted"))                          # -> 'rate_limit'

Observed with a fake client:

Error (status 400) kind calls made sleeps
max_tokens=5000 exceeds limit 4096 overloaded 4 2s, 4s, 8s
n must be <= 128, got 1500 overloaded 4 2s, 4s, 8s
user id 6429 not permitted rate_limit 4 60s, 60s, 60s (180s total)
plain BadRequestError / ContentPolicyViolationError unknown 3 1s, 2s
ValueError / TypeError / KeyError unknown 3 1s, 2s

A client error that can never succeed ties up an agent for up to 3 minutes and bills 4 requests.

Fix: use status_code (or the exception class) first, and only fall back to message matching when no status is available.

status = getattr(error, "status_code", None) or getattr(getattr(error, "response", None), "status_code", None)
if isinstance(status, int):
    if status == 429: return "rate_limit"
    if status in (500, 502, 503, 504, 529): return "overloaded"
    if status in (400, 404, 413, 422): return "non_retryable"   # new kind: fail fast
# ... existing substring heuristics only when status is None,
# using word boundaries: re.search(r"\b429\b", error_str)

Programming errors (ValueError, TypeError, KeyError) should not enter the retry loop.


2. Workflow steps retry every exception 3 times by default, including non-idempotent function steps

Where: workflows/workflows.py:3360-3380, with the same pattern near :1930-1936 and :2113

max_retries = getattr(step, 'max_retries', 3)
...
except Exception as e:
    step_error = e
    is_retryable = getattr(e, 'is_retryable', True)   # <- default True for everything
    if not is_retryable:
        break
    retry_count += 1
    if retry_count <= max_retries:
        self._retry_backoff(...)
        continue

Any exception raised by user code, such as a ValueError from a bug, is replayed 3 more times with backoff. A function step that sends an email, charges a card or inserts a row runs 4 times.

Reproduction: Workflow(steps=[bad]) where bad raises ValueError:

  • 4 calls
  • 7.0s elapsed
  • retries: 4

The same 4x replay happens inside parallel([...]) branches.

Fix: make retries opt-in for deterministic failures and function steps, and retry only transient errors for agent steps.

_TRANSIENT = (TimeoutError, ConnectionError)   # plus LLM errors classified as rate_limit/overloaded (see #1)

def _should_retry(step, e) -> bool:
    explicit = getattr(e, 'is_retryable', None)
    if explicit is not None:
        return explicit
    if getattr(step, 'retry_on_error', False):   # opt-in for arbitrary exceptions
        return True
    return isinstance(e, _TRANSIENT)

Default max_retries for plain callables should be 0.


3. AgentTeam.wait_for_completions() returns nothing for fast sub-agents, and _completion_events grows without bound

Where: agents/agents.py:4461 (async variant :4678), cleanup at :4515-4519, append at :4424 (async :4635)

# wait_for_completions
with self._spawn_lock:
    target_agents = agent_ids or list(self._spawned_agents.keys())   # finished agents are already gone
...
# _handle_sub_agent_completion
if agent_id in self._spawned_agents:
    del self._spawned_agents[agent_id]                                # deleted on completion
...
self._completion_events.append(completion_event)                      # plain list, never trimmed

Completion removes the spawn record, so the default target set (agent_ids omitted) only contains agents still running. If every sub-agent already finished, target_agents is empty and the wait returns [] immediately. The async variant derives its targets the same way.

Reproduction: spawn 3 instant sub-agents and sleep 1s.

  • len(_spawned_agents) is 0 and len(_completion_events) is 3.
  • wait_for_completions(timeout=2) returns 0 results (expected 3).
  • With explicit agent_ids it returns 3.
  • After 2000 announce_completion(..., 'R'*1000) calls, len(_completion_events) is 2003, with each full result string retained.

Fix: derive default targets from both live spawns and recorded events, and bound the event store.

with self._spawn_lock:
    target_agents = agent_ids or (
        set(self._spawned_agents) | {e.agent_id for e in self._completion_events}
    )

# in __init__:
self._completion_events = collections.deque(maxlen=self._max_completion_events)   # e.g. 256

Alternatively, drain events once they are returned from wait_for_completions.


Also found, lower priority

  • memory/file_memory.py:544-567: _auto_promote_to_long_term bypasses long_term_limit, rebuilds existing_ids per item, and rewrites the whole JSON file on every add. 1500 high-importance adds produced 1500 long-term entries (limit 1000) and took 23.6s.
  • llm/rate_limiter.py:167-218: acquire_tokens(n) with n above the bucket capacity never succeeds in the sync path, and silently goes negative in the async path. Nothing in the SDK calls it today.
  • memory/mcp_config.py:289,348, memory/hooks.py:417, memory/rules_manager.py:824, memory/docs_manager.py:390 use non-atomic write_text, unlike file_memory, session/store and storage/base, which use temp file plus os.replace.

Acceptance criteria

  • A 400, 404 or 422 response, and programming errors such as TypeError, fail on the first attempt with no retry sleeps.
  • A workflow function step that raises a non-transient exception runs once by default.
  • wait_for_completions() with no arguments returns results for sub-agents that completed before the call.
  • _completion_events has a bounded size.
Ngôn ngữ chính
Python
Star
9.3k
Fork
1.5k
Merge trung bình
1 ngày 5 giờ
Pull request đã merge (30 ngày)
520

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của MervinPraison/PraisonAI

Tất cả issue của MervinPraison/PraisonAI

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.