fix(worker): delay/approval 到期触发失败仍 ZREM 出排程——最后一跳 at-most-once,Redis 瞬断=挂起执行永久失醒
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python, redis
- Domain
- backend, distributed-systems
Research direction
Start with publish_resume_event in base_service.py:379-400 and _run_scheduled_task/_finalize_scheduled in delay_service.py:224-261. Reproduce a publish failure with a mock, then trace the retry and removal behavior. Done means the task remains in the Redis ZSET after failure and a later retry succeeds, with the chosen retry or dead-letter behavior covered by tests.
Written by the indexing model from the issue text.
Description
场景 gap 单(okguitar 提报)· 优先级 P1 · 依赖:无
基线 HEAD = ddcedae。
Summary
base_service.py:379-400publish_resume_event捕获一切异常只logger.error(「发布失败只告警,不打断任务主流程」)。delay_service.py:224-249_run_scheduled_task:handle_task正常返回(触发失败也返回 True,因为异常被上游吞了)→ finally →_finalize_scheduled(member)→ ZREM 出排程 ZSET(:251-261)。失败与成功出队路径完全相同,无重试、无死信。- 补偿前提不成立:EventReconciler 只能扫到「已落事件存储」的事件——Redis 整体瞬断时落盘与直发一起失败,事件存储里没有东西可回扫(
_persist_resume_event_best_effort同样只 warning)。
影响
C4-2 把「等待期」做崩溃安全了,但「触发瞬间」仍是 at-most-once:到期提交后恰逢 Redis 抖动 → publish 失败被吞、任务已 ZREM——挂起执行的唯一唤醒凭据消失,执行永久 suspended,且没有计数器/日志键能系统性发现。approval/http_callback 复用同一 publish_resume_event,同病。对比:任务队列本体是 at-least-once + DLQ,唯独这最后一跳退化成 fire-once。
建议
publish_resume_event 异常上抛,_run_scheduled_task 异常路径不 ZREM(留 ZSET 下轮重试,配重试计数/退避,超限转显式死信键);或触发前写 intent 键、确认后删除,启动时对账孤儿 intent。
验收:mock publish 抛异常,断言任务仍在 ZSET 且下轮重试成功。
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 4h 24m
- Merged PRs (30d)
- 4
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
webrecorder/warcio#209 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Jeffy123-zhu/logspecter#20 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 1 day