"Blocking found group but reserve failed" errors on op-worker container plus high CPU usage
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 冷清
- 技术栈
- redis, typescript
调研方向
首先检查 op-worker 对 Redis groupmq 的处理，以及包含 "Blocking found group but reserve failed." 的日志。调查 group、lock 或 job 在容器和服务器重启后仍然卡住的方式，然后验证 worker 能够恢复，且不会快速重复该错误或导致日志无限增长。
由索引模型根据 Issue 内容生成。
描述
Hello. First of all, thanks for such a nice application! I've been testing OpenPanel with a few online stores that I host and it's really good.
But today I found a rather unusual problem with my setup. Suddenly, CPU usage on the server I'm running OpenPanel jumped from 6% to 62%. Since I'm working on some integrations I thought maybe I did something wrong, but with help from AI I nailed down what happened.
Somehow the op-worker container (Redis) got stuck with "Blocking found group but reserve failed: group=06da2f33f2dda361f48d404fb2066f43 (reserve took 0ms)" being printed to the log quite rapidly. I don't know much about Redis but AI helped me through it. Somehow this key group got stuck or corrupt or something (I still don't know how, though). This key group was seemingly being used by one of the sites I added to OpenPanel, so in the end I had to wipe up this key group and do some other cleanup.
It first told me to check what's up.
LLEN groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43:active
ZCARD groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43
ZRANGE groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43 0 -1 WITHSCORES
GET groupmq:group_events🔒06da2f33f2dda361f48d404fb2066f43
ZSCORE groupmq:group_events:ready 06da2f33f2dda361f48d404fb2066f43
Then ZRANGE returned two jobs, I had to run these on them:
HGETALL groupmq:group_events:job:<JOB_ID>
EXISTS groupmq:group_events:processing:<JOB_ID>
Then I had to unblock the key group:
MULTI
ZREM groupmq:group_events:ready 06da2f33f2dda361f48d404fb2066f43
DEL groupmq:group_events🔒06da2f33f2dda361f48d404fb2066f43
DEL groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43:active
EXEC
Then I had to remove those jobs:
- for each <JOB_ID>:
ZREM groupmq:group_events:processing <JOB_ID>
DEL groupmq:group_events:processing:<JOB_ID>
DEL groupmq:group_events:job:<JOB_ID>
DEL groupmq:group_events:g:06da2f33f2dda361f48d404fb2066f43
After that, the stuck key was cleared and everything went back to normal. Before this I tried redeploying OpenPanel, restarted the container, rebooted the server, nothing helped. Clearing that stuck key is what helped. I also had to clear the log files as they grew to over 2 GB.
One thing I found odd though, is (and correct me if I'm wrong) but Redis is an in memory cache, maybe it should have cleared this by itself after a restart?
In any case, this happened to me and I'm posting it here in hopes it could help someone.
- 主要语言
- TypeScript
- 星标
- 7.1k
- 派生
- 516
- 平均合并
- 34 分钟
- 30 天内合并 PR
- 3
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
Openpanel-dev/openpanel 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 68/100
Openpanel-dev/openpanel#543 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 74/100
Openpanel-dev/openpanel#541 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 88/100
Openpanel-dev/openpanel#532 ·
维护者通常 1 天内回复
-
Period comparison badge shows wrong percentage for decreases (100 → 50 shows ↓100%)可能已有人在做 @sarmah-rup 于 17 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
Openpanel-dev/openpanel#526 ·
维护者通常 1 天内回复
-
Self-hosted missing op1-replay.js可能已有人在做 @houstona 于 18 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
Openpanel-dev/openpanel#512 · 1 条评论 ·
维护者通常 1 天内回复
查看 Openpanel-dev/openpanel 的全部 Issue
相似的 Issue
-
bnv-manager-v2: accept_invite creates the Mailcow domain admin before the commit可能已有人在做 @myOmikron 今天认领。 未关闭bug ready-for-agent
难度 2/5 1-3 小时 新手友好度 82/100
-
platform:windows tool triaged wontfix
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
nicolettas-muggelbude/RechnungsFee#448 ·
维护者通常 1 天内回复
-
Android bug feature/error-tracking react-native team/error-tracking
难度 2/5 1-3 小时 新手友好度 72/100
PostHog/posthog-js#5310 · 1 条评论 ·
维护者通常 1 天内回复