Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

A streaming group by on a text key hands out ordinals on one thread

Open
  • Is something wrong?
#997 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data-engineering, performance

Research direction

The issue is about optimizing a streaming group by for text keys in a columnar engine. Start by examining the current serial pass code for key mapping and ordinal assignment, likely in the group by operator. Run the ClickBench q33 query to measure baseline performance. Research the two proposed designs: per-worker maps with a merge versus maintaining first-seen order. The goal is to match or beat the 50.6 ms benchmark while preserving correctness across all tests.

Written by the indexing model from the issue text.

Description

area/exec area/kernel performance

Left over from #991, which split the text key map into three passes and got two of them onto the cores. The third one is still serial and this is the issue for it.

What the serial pass does: for every row of a chunk whose key the map did not already hold, probe the table, compare the bytes on a hash match, and if the key really is new, append its bytes to the builder and give it the next ordinal. It has to run in row order because the ordinal a key gets is the position of the row that first carried it, and a group by result here comes out in the order its keys were first seen.

q33 is the worst case for it in ClickBench. SELECT URL, COUNT(*) AS c FROM hits GROUP BY URL ORDER BY c DESC LIMIT 10 has no filter under it, so all 1,000,000 rows reach the operator and 275,494 of them carry a URL nobody has seen yet. About half the rows miss the parallel lookup and land in the serial pass, which caps the operator near two cores. It measures at 1.43.

Where it stands: q33 through the planner at 1M is 74.3 ms against the 50.6 ms it cost before #986 introduced the map, measured on a pair quiet enough to compare. Every other text group by in the suite is better than it was, because a filter under the operator cuts the rows that reach it and the serial pass shrinks with them.

One thing that looked like it might close the gap without a design change does not. Growing the table by four instead of two cuts the rebuilds from ten to five and the live slots moved from about 524,000 to about 175,000, all of it inside the serial pass, and it measures as nothing. The rebuild was never the big term. The two costs are half a million random probes into a table growing to four megabytes and the bytes of 275,494 keys going into the builder. Numbers are on #991.

Two ways out that I can see, and neither is small, which is why this is its own issue.

Give up first seen order and hand out ordinals per worker, then merge the per worker maps at the end. That is what the stacking merge did in effect, and it is why the stacking merge got its parallelism for free. It means a result comes out in some other order and either something downstream sorts or the contract changes.

Keep first seen order and pay to restore it. Workers each build a local map, and the merge gives final ordinals by the smallest row index each key was seen at, which is a sort over the distinct keys rather than over the rows. On q33 that is 275,494 against 1,000,000 and it may pay. On a query where nearly every row is its own group it does not.

  • Pick between per worker maps with a merge and staying serial, with the cost of the merge measured rather than argued
  • q33 through the planner is at or under the 50.6 ms it cost before #986
  • The text group bys that improved in #986 and #992 keep their improvement, and q9 on an int key does not move
  • Every ClickBench answer is unchanged, and the pipeline, chunk agreement and SQL tests pass
Dominant language
Mojo
Stars
3
Forks
0
Avg merge
9m
Merged PRs (30d)
407

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tamnd/firepanda

All issues in tamnd/firepanda

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.