[server] Periodic KV snapshot initialization races with Replica deletion
维护者通常 1 天内回复
@MaheshS08 已经在做这个了。
开始于 2026年10月13日。
评估
调研方向
Start with Replica.delete() and Replica.dropKv(), then trace how closeableRegistryForKv closes PeriodicSnapshotManager and how snapshot initialization runs under kvLock. Use the deterministic regression described in the issue with the existing ReplicaTest fixtures to reproduce the registry-closing window. Done means deletion stops or synchronizes periodic snapshot initialization before the KV registry closes, with the regression confirming no closed-registry IOException is logged.
由索引模型根据 Issue 内容生成。
描述
Search before asking
- I searched in the issues and found nothing similar.
Fluss version
main (development)
Please describe the bug 🐞
Periodic KV snapshot initialization can race with Replica deletion and attempt to register resources after the Replica's KV registry has been closed.
java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument.
at AbstractAutoCloseableRegistry.registerCloseable
at AsyncSnapshotCallable$AsyncSnapshotTask.<init>
at AsyncSnapshotCallable.toAsyncSnapshotFutureTask
at SnapshotRunner.snapshot
at KvTabletSnapshotTarget.initSnapshot
at PeriodicSnapshotManager.lambda$triggerSnapshot$2
Replica.dropKv() closes closeableRegistryForKv. The registry first sets closed=true, then closes its registered resources, including PeriodicSnapshotManager. A scheduled snapshot can still run between these steps and reach task registration while the registry is already closed.
The manager's close() also does not serialize with snapshot initialization under the tablet's kvLock, so cancelling the scheduled future alone does not cover an initialization already in progress.
Deterministic reproduction
A local regression uses the existing ReplicaTest fixtures and real Replica, RocksDB, snapshot target, runner and registry:
- Create a KV leader, write and flush records, and retain the snapshot timer automatically scheduled during leader initialization.
- Register a latch-controlled Closeable after the snapshot manager. Call
Replica.delete()on another thread and pause resource closing after the registry becomes closed, before the manager stops. - Execute the original scheduled timer, then release the latch and wait for deletion to finish.
This enlarges a valid closing window without mocking the initialization exception. The assertion that no closed-registry IOException is logged fails on both revisions:
| Regression | Base b530fcc2 |
PR #4492 19870c605 |
|---|---|---|
| Retry after snapshot-init failure | Fails: no replacement task | Passes |
| Snapshot initialization during Replica deletion | Fails: closed-registry IOException | Fails: same IOException |
Four occurrences were observed during test table deletion. Deletion completed, and no snapshot corruption was established.
Expected behavior
Replica deletion should stop or cancel periodic snapshots cleanly, without allowing snapshot task registration after the KV registry closes.
Solution
Serialize stopping the snapshot manager with snapshot initialization before closing the KV registry and dropping the tablet, while preserving cancellation and cleanup of already registered tasks.
Related: #4489 and #4492 address retry after initialization failure. The closing race exists before that PR and requires a separate lifecycle fix.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- 主要语言
- Java
- 星标
- 2.2k
- 派生
- 658
- 平均合并
- 2 天 23 小时
- 30 天内合并 PR
- 101
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/fluss 的其他 Issue
-
[lake/iceberg] Reserved write-mode properties can be set and reset through their `iceberg.` key可能已有人在做 @aphenix-r7 于 9 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
[common] Decimal#hashCode is inconsistent with Decimal#equals可能已有人在做 @firasbouzazi 于 20 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
[security] PlainSaslServerProvider conflicts with Hadoop SASL provider causing authentication failure可能已有人在做 @xinzhang-hello 于 26 天前认领。 未关闭
难度 1/5 1 小时以内 新手友好度 88/100
维护者通常 1 天内回复
-
[server] KV snapshot scheduling stops permanently after a transient snapshot-init failure可能已有人在做 @zhang-arvin 于 26 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 88/100
维护者通常 1 天内回复
-
[docs] Document advertised-resource and ACL principal ignore-case options可能已有人在做 @EchoLee5 于 38 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 85/100
-
[Bug]: REST API URL starting with "http" but no scheme (e.g. httpbin.org/get) is not given an http:// prefix可能已有人在做 @ItsDeadlyProgrammer 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 68/100
appsmithorg/appsmith#42471 · 1 条评论 ·
维护者通常 2 天内回复
-
code-maintenance
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
-
bug priority: medium Review effort [1-5]: 3 status: needs-triage type: bug
难度 2/5 1-3 小时 新手友好度 82/100
carlos-emr/carlos#5258 ·
维护者通常 1 天内回复