[server] KV snapshot scheduling stops permanently after a transient snapshot-init failure
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 88/100
Research direction
Start in fluss-server/.../kv/snapshot/PeriodicSnapshotManager.java at triggerSnapshot(), then compare the initialization-failure path with the existing async failure handling and scheduleNextSnapshot() flow. Verify that a transient init failure does not end the one-shot chain and that snapshot attempts resume after the backend returns.
Written by the indexing model from the issue text.
Description
Search before asking
- I searched in the issues and found nothing similar.
Fluss version
main (development)
Please describe the bug
I searched in the issues and found nothing similar. Closest family:
- #2977
- #2978, A different start-time trigger, already fixed.
Motivation
A transient backend outage during snapshot initialization permanently disables KV snapshots for the affected buckets until the TabletServer restarts. The failure is silent after a single ERROR line: no retry, no metric alarm, no recovery when the backend returns.
Reproduction
- Fluss 1.0.0 (also present on
main, file is identical), 1 coordinator + 2 tablet servers,kv.snapshot.interval: 60s, S3-compatible remote storage. - Stop the object store for ~6 minutes, then restore it.
- During the outage, tablets log
Connection refusedwith AWS SDK retries; writes keep flowing (local hot path, unaffected). - After the backend returns: no new snapshot for 14+ minutes (~14 missed 60s cycles). Only
snap-1/snap-2remain (retention keeps the last two). - Restarting one tablet produces a fresh snapshot ~50s later (
snap-4), confirming the scheduler — not the backend path — was dead.
Root cause
fluss-server/.../kv/snapshot/PeriodicSnapshotManager.java, triggerSnapshot() (v1.0.0 and main identical):
try {
snapshotRunnableOptional = target.initSnapshot();
} catch (Exception e) {
LOG.error("Fail to init snapshot during triggering snapshot.", e);
return; // <-- scheduleNextSnapshot() never called
}
Scheduling is a one-shot chain (schedule(triggerSnapshot, delay) + explicit scheduleNextSnapshot()), so returning here without rescheduling breaks the chain permanently for that bucket. The async-phase failure path (whenComplete) does track retries; only the init path drops the chain.
Observed log line proving the trigger:
ERROR org.apache.fluss.server.kv.snapshot.PeriodicSnapshotManager [] - Fail to init snapshot during triggering snapshot.
followed by total silence (no further attempts, successes, or failures).
Solution
Call scheduleNextSnapshot() on the init-failure path too (ideally with backoff after consecutive failures, mirroring the existing numberOfConsecutiveFailures tracking in the async path), so a transient backend outage degrades to missed snapshots instead of a permanently dead scheduler.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- Dominant language
- Java
- Stars
- 2.2k
- Forks
- 656
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 108
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/fluss
-
[CI] NettyClientTest.testServerDisconnection may fail with DisconnectExceptionPossibly taken @yangshangqing95 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
[lake/iceberg] Reserved write-mode properties can be set and reset through their `iceberg.` keyPossibly taken @aphenix-r7 claimed this 3 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/fluss#4658 · 1 comment ·
Maintainers usually reply within 1 day
-
[common] Decimal#hashCode is inconsistent with Decimal#equalsPossibly taken @firasbouzazi claimed this 14 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/fluss#4538 · 1 comment ·
Maintainers usually reply within 1 day
-
[security] PlainSaslServerProvider conflicts with Hadoop SASL provider causing authentication failurePossibly taken @xinzhang-hello claimed this 20 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
[docs] Document advertised-resource and ACL principal ignore-case optionsPossibly taken @EchoLee5 claimed this 32 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
Similar issues
-
enhancement Improvement
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
zowe/zowe-client-java-sdk#647 ·
-
NullPointerException when a `DeserializationProblemHandler` returns `null` for `byte[]` (single value unwrapped) and `char[]` elementsPossibly taken @Marti2203 claimed this today. Open
Difficulty 2/5 Half a day Newbie friendliness 88/100
FasterXML/jackson-databind#6300 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
MustacheCase/zanadir#254 ·
Maintainers usually reply within 1 day
-
content good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
cpinitiative/usaco-guide#6699 ·
Maintainers usually reply within 1 day
-
BC remittance upload: a claim MSP refused (S03) is saved with a blank status instead of refused (F)Possibly taken A pull request linked to this issue is open or already merged. Opentype: bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
carlos-emr/carlos#4933 ·
Maintainers usually reply within 1 day