[FLINK] Legacy group window aggregation ignores the window size, slide and type

Open
#13,717 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, java, scala

Research direction

Start with the Gluten copy of StreamExecGroupWindowAggregate, which passes hard-coded size, slide, offset and window type instead of reading the LogicalWindow, and with GroupWindowAggregationNode. The native side is StatefulPlanner::transformGroupWindowAggregationOperator in the velox4j pinned commit, which always builds SessionWindowAssigner(10). Done means the parameters reach both places or the translation fails with a clear not-supported error, and a test checks tumble, hop and session results, not just that the job runs.

Written by the indexing model from the issue text.

Description

A legacy group window query (GROUP BY TUMBLE(...), HOP(...) or SESSION(...)) is translated by the Gluten copy of StreamExecGroupWindowAggregate, which never reads the LogicalWindow. It passes size, slide and offset 0 and window type 1 to StreamWindowPartitionFunctionSpec, and isEventTime = true and window type 1 to GroupWindowAggregationNode (all marked TODO).

The native side doesn't read them either. In bigo-sg/velox at the commit velox4j pins (5c907ef6), StatefulPlanner::transformGroupWindowAggregationOperator builds the SliceAssigner with size, step and offset 0 and always uses SessionWindowAssigner(10, ...) ("TODO: support window parameters"). So every legacy group window, tumbling or hopping included, becomes a session window with a gap of 10, whatever the query asks for, and the job reports no error. Nexmark q11 (SESSION(dateTime, INTERVAL '10' SECOND)) runs this path in NexmarkTest, but its sink is a blackhole, so the results aren't checked.

Window TVF queries (TUMBLE(TABLE ...)) take a different path that does read the window parameters (WindowUtils.extractWindowParameters).

A complete fix needs the parameters in both places. Until the native operator supports them, failing the translation with a clear "not supported" error, or leaving these queries on Flink's own operator, would avoid wrong results. I'd like to hear which the gluten-flink maintainers prefer before sending a patch.

This issue was written with the assistance of AI (Claude Opus).

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.