Tracking: establish at-least-once delivery for sink connectors
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 25/100
- Issue 类型
- 功能
- 描述清晰度
- 需要澄清
- 活跃度
- 活跃
- 技术栈
- rust
调研方向
Start with the proposed work items #4233–#4238 and the local HTTP regression sequence described here. Read #2928, discussions #3203 and #3518, and PRs #3954 and #4152 to understand existing decisions and work. Done requires an agreed contract, linked implementation and recovery tests, supported configurations, and migration and operational documentation.
由索引模型根据 Issue 内容生成。
描述
I propose tracking the remaining sink delivery work under one agreed contract, with separate implementation issues and recovery tests as acceptance criteria.
The runtime can save a consumer offset before the sink delivers the corresponding records. If delivery fails, restarting with the same consumer group can skip those records. They may remain in Iggy's retained log, but normal recovery does not deliver them.
A local HTTP regression test reproduces this sequence: the endpoint rejects a record with HTTP 503, the runtime restarts with the same consumer group after the endpoint recovers, and a later record arrives while the rejected record does not. This test covers rejection followed by restart; abrupt termination needs additional coverage.
Issue #2928 and discussions #3203 and #3518 already describe the underlying problem and proposed approaches. PR #3954, by kriti-sc, adds control over commit timing and provides implementation work to build on. PR #4152 has since improved failure reporting and accounting, but explicitly does not introduce batch replay. The status of #2927 should be reconciled with that merged work.
The proposed contract is:
For each partition, the runtime advances the recovery checkpoint only after every record it passes has satisfied the configured delivery contract, including all required destination writes. Failed or uncertain delivery remains recoverable. Recovery may produce duplicates.
Acceptance into a plugin's memory buffer must be distinct from durable delivery. Unexpected decode, transform, and serialization failures must remain unresolved until recovered or handled by an explicit policy. A configured filter is a separate outcome. If a dead-letter policy is supported, its write must succeed before the checkpoint advances, and the resulting guarantee must name that alternative destination.
Each supported sink configuration must define what its delivery acknowledgment proves. The contract also depends on source durability, sufficient retention, and eventual recovery. Missing replay data must produce a visible recovery failure.
Proposed work:
- #4233: Define the sink delivery and recovery contract.
- #4234: Implement explicit sink delivery acknowledgments and safe checkpoint advancement.
- #4235: Preserve failed records throughout the sink pipeline.
- #4236: Migrate sink plugins and document supported configurations.
- #4237: Add a shared recovery conformance suite.
- #4238: Introduce safe defaults, migration guidance, and operational visibility.
Existing issues and PRs should be linked to these work items where their scope matches. The test suite should begin alongside the contract work, with regression coverage included in each implementation change.
This tracker is complete when the agreed delivery contract is enforced for every configuration advertised as supporting it, incompatible configurations fail explicitly, required recovery tests pass in CI, and the migration and operational documentation are available.
Maintainer feedback is requested on the target contract, the initial set of supported sinks, the interface migration, and how to continue the work in #3954.
- 主要语言
- Rust
- 星标
- 5.2k
- 派生
- 502
- 平均合并
- 3 天 11 小时
- 30 天内合并 PR
- 143
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/iggy 的其他 Issue
-
go good first issue
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
HTTP client treats a consumer group as a plain consumer可能已有人在做 @mohitduhan19 于 18 天前认领。 未关闭bug rust server
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
Add the fail-fast connection option to the Rust and Python quick-starts可能已有人在做 @QIW9 于 5 天前认领。 未关闭docs good first issue
难度 1/5 1-3 小时 新手友好度 92/100
维护者通常 1 天内回复
-
bug python sdk
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
-
good first issue python
难度 3/5 1-2 天 新手友好度 72/100
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
libnativeapi/nativeapi#21 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
longbridge/nsh#804 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复