[FLINK] Legacy group window aggregation ignores the window size, slide and type
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 22/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- stream-processing
Research direction
Start with the Gluten copy of StreamExecGroupWindowAggregate, which passes hard-coded size, slide, offset and window type instead of reading the LogicalWindow, and with GroupWindowAggregationNode. The native side is StatefulPlanner::transformGroupWindowAggregationOperator in the velox4j pinned commit, which always builds SessionWindowAssigner(10). Done means the parameters reach both places or the translation fails with a clear not-supported error, and a test checks tumble, hop and session results, not just that the job runs.
Written by the indexing model from the issue text.
Description
A legacy group window query (GROUP BY TUMBLE(...), HOP(...) or SESSION(...)) is translated by the Gluten copy of StreamExecGroupWindowAggregate, which never reads the LogicalWindow. It passes size, slide and offset 0 and window type 1 to StreamWindowPartitionFunctionSpec, and isEventTime = true and window type 1 to GroupWindowAggregationNode (all marked TODO).
The native side doesn't read them either. In bigo-sg/velox at the commit velox4j pins (5c907ef6), StatefulPlanner::transformGroupWindowAggregationOperator builds the SliceAssigner with size, step and offset 0 and always uses SessionWindowAssigner(10, ...) ("TODO: support window parameters"). So every legacy group window, tumbling or hopping included, becomes a session window with a gap of 10, whatever the query asks for, and the job reports no error. Nexmark q11 (SESSION(dateTime, INTERVAL '10' SECOND)) runs this path in NexmarkTest, but its sink is a blackhole, so the results aren't checked.
Window TVF queries (TUMBLE(TABLE ...)) take a different path that does read the window parameters (WindowUtils.extractWindowParameters).
A complete fix needs the parameters in both places. Until the native operator supports them, failing the translation with a clear "not supported" error, or leaving these queries on Flink's own operator, would avoid wrong results. I'd like to hear which the gluten-flink maintainers prefer before sending a patch.
This issue was written with the assistance of AI (Claude Opus).
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day