Kafka Streams runner: bundles are not bounded by time
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 活跃
- 技术栈
- java, kafka
调研方向
从 MetricsAcrossBundlesTest 以及从记录处理或 punctuator 关闭 Fn-API bundles 的 Kafka Streams runner 路径开始。检查 StreamTask#process() 和修正后的 commit 计账，然后在四个分区上复现两个串联 GroupByKeys 的情况。完成标准是：maxBundleTimeMs 在不产生重复输出的情况下限制稀疏流 bundles，并使用数据驱动的 bundle markers，而不是 wall-clock punctuator。
由索引模型根据 Issue 内容生成。
描述
--maxBundleTimeMs is accepted but has no effect. A bundle is closed when it reaches --maxBundleSize elements or when a watermark arrives, so on a sparse stream the elements already fed to a bundle are not emitted until the next watermark, and the watermark cannot advance past them.
The natural implementation — closing the bundle from a wall-clock punctuator — produces duplicate output against a real broker. With it enabled, an integration test running two chained GroupByKeys across four partitions emits its single group six times, reproducibly, and the count keeps climbing after the input has stopped.
What has been ruled out:
- Metrics folding. Splitting the same input across many bundles does not change a user counter (
MetricsAcrossBundlesTest). ProcessorContext.commit(). The duplication happens with the commit request removed, and does not happen with the punctuator disabled but the commit still requested.
So it appears to be closing a Fn-API bundle from a punctuator rather than from record processing.
Note that Kafka Streams sets commitOffsetNeeded inside StreamTask#process(), so data produced from a punctuator sits outside that accounting; KAFKA-6906 was a bug of that shape.
That note was wrong — see the comment below. There is no commitOffsetNeeded flag; it is commitNeeded, and it is set after a punctuation runs as well as from process(), so punctuator output is not outside the commit accounting. Corrected by Matthias J. Sax on the Kafka dev list. The cause of the duplication is still unknown, and the way forward is data-driven bundle markers rather than a wall-clock punctuator.
Part of #18479.
- 主要语言
- Java
- 星标
- 8.7k
- 派生
- 4.7k
- 平均合并
- 2 天 14 小时
- 30 天内合并 PR
- 271
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/beam 的其他 Issue
-
[Bug]: Row.toString throws for an ITERABLE field that is not backed by a List可能已有人在做 @PDGGK 于 61 天前认领。 未关闭java P3
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
apache/beam#39624 · 2 个 reaction ·
维护者通常 1 天内回复
-
[Bug]: PubsubIO used in batch incorrect batch cutoff size可能已有人在做 @1fanwang 于 50 天前认领。 未关闭bug io P3 pinned pubsub
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
build P3 sub-task
难度 1/5 1 小时以内 新手友好度 68/100
维护者通常 1 天内回复
-
bug gcp io java P3
难度 1/5 1 小时以内 新手友好度 65/100
维护者通常 1 天内回复
相似的 Issue
-
RRateLimiter acquire methods accept negative permits and raise available permits above the rate.未关闭
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 4 天内回复
-
难度 1/5 1 小时以内 新手友好度 75/100
krahets/hello-algo#1993 ·
-
难度 2/5 1-3 小时 新手友好度 62/100
NationalSecurityAgency/ghidra#9884 ·
维护者通常 2 天内回复
-
Status: Pending Review Type: Feature Request
难度 1/5 1 小时以内 新手友好度 82/100
OpenRefine/OpenRefine#8019 ·
维护者通常 1 天内回复
-
[TEMP] pin maven version to 3.9.x可能已有人在做 @petrenyi-mark 今天认领。 未关闭
难度 2/5 1 小时以内 新手友好度 62/100