[server] KV snapshot scheduling stops permanently after a transient snapshot-init failure
维护者通常 1 天内回复
评估
调研方向
从 fluss-server/.../kv/snapshot/PeriodicSnapshotManager.java 中的 triggerSnapshot() 开始，然后将初始化失败路径与现有的异步失败处理和 scheduleNextSnapshot() 流程进行比较。确认暂时性的初始化失败不会结束 one-shot 链，并且在后端恢复后会继续尝试执行 snapshot。
由索引模型根据 Issue 内容生成。
描述
Search before asking
- I searched in the issues and found nothing similar.
Fluss version
main (development)
Please describe the bug
I searched in the issues and found nothing similar. Closest family:
- #2977
- #2978, A different start-time trigger, already fixed.
Motivation
A transient backend outage during snapshot initialization permanently disables KV snapshots for the affected buckets until the TabletServer restarts. The failure is silent after a single ERROR line: no retry, no metric alarm, no recovery when the backend returns.
Reproduction
- Fluss 1.0.0 (also present on
main, file is identical), 1 coordinator + 2 tablet servers,kv.snapshot.interval: 60s, S3-compatible remote storage. - Stop the object store for ~6 minutes, then restore it.
- During the outage, tablets log
Connection refusedwith AWS SDK retries; writes keep flowing (local hot path, unaffected). - After the backend returns: no new snapshot for 14+ minutes (~14 missed 60s cycles). Only
snap-1/snap-2remain (retention keeps the last two). - Restarting one tablet produces a fresh snapshot ~50s later (
snap-4), confirming the scheduler — not the backend path — was dead.
Root cause
fluss-server/.../kv/snapshot/PeriodicSnapshotManager.java, triggerSnapshot() (v1.0.0 and main identical):
try {
snapshotRunnableOptional = target.initSnapshot();
} catch (Exception e) {
LOG.error("Fail to init snapshot during triggering snapshot.", e);
return; // <-- scheduleNextSnapshot() never called
}
Scheduling is a one-shot chain (schedule(triggerSnapshot, delay) + explicit scheduleNextSnapshot()), so returning here without rescheduling breaks the chain permanently for that bucket. The async-phase failure path (whenComplete) does track retries; only the init path drops the chain.
Observed log line proving the trigger:
ERROR org.apache.fluss.server.kv.snapshot.PeriodicSnapshotManager [] - Fail to init snapshot during triggering snapshot.
followed by total silence (no further attempts, successes, or failures).
Solution
Call scheduleNextSnapshot() on the init-failure path too (ideally with backoff after consecutive failures, mirroring the existing numberOfConsecutiveFailures tracking in the async path), so a transient backend outage degrades to missed snapshots instead of a permanently dead scheduler.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- 主要语言
- Java
- 星标
- 2.2k
- 派生
- 656
- 平均合并
- 1 天 21 小时
- 30 天内合并 PR
- 127
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/fluss 的其他 Issue
-
[lake/iceberg] Reserved write-mode properties can be set and reset through their `iceberg.` key可能已有人在做 @aphenix-r7 于 1 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
[common] Decimal#hashCode is inconsistent with Decimal#equals可能已有人在做 @firasbouzazi 于 13 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
[security] PlainSaslServerProvider conflicts with Hadoop SASL provider causing authentication failure可能已有人在做 @xinzhang-hello 于 18 天前认领。 未关闭
难度 1/5 1 小时以内 新手友好度 88/100
维护者通常 1 天内回复
-
[docs] Document advertised-resource and ACL principal ignore-case options可能已有人在做 @EchoLee5 于 30 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
-
[client] LimitBatchScanner leaks Arrow read context on log-table LIMIT scan可能已有人在做 @EchoLee5 于 30 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
相似的 Issue
-
agent:working documentation enhancement go:build lane:needs-decision
难度 2/5 1-3 小时 新手友好度 72/100
Sanne/incus-spawn#1435 · 3 条评论 ·
维护者通常 1 天内回复
-
enhancement good first issue up-for-grabs
难度 2/5 1-3 小时 新手友好度 80/100
AnikethTS/ChangeRadar#22 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
openai/openai-java#1202 ·
维护者通常 1 天内回复
-
admin sql type: bug
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 2 天内回复
-
library-new-request priority
难度 2/5 1-3 小时 新手友好度 70/100
oracle/graalvm-reachability-metadata#10911 · 1 条评论 ·
维护者通常 1 天内回复