[Feature Request]: Bound PGBKCVOperation precombine table by in-memory size, not key count
Maintainers usually reply within 1 day
Nobody has claimed this yet.
- #39767 by @Dharshika-11 — closed without merging
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- data-engineering
Research direction
Start in sdks/python/apache_beam/runners/worker/operations.py at PGBKCVOperation and its TODO about bounding by in-memory size. Compare the sibling PGBKOperation and inspect how the Count, Mean, min, max, and sum accumulators are represented. Done means the precombine table is bounded using an in-memory-size estimate rather than relying only on key count, while accounting for wrapped or native accumulators.
Written by the indexing model from the issue text.
Description
What would you like to happen?
PGBKCVOperation (sdks/python/apache_beam/runners/worker/operations.py) caps its precombine table by key count (max_keys, default 100k / 1M for Count/Mean/min/max/sum), flushing ~10% of keys once the count is hit. The code already flags this as a stopgap:
# TODO(b/36567833): Bound by in-memory size rather than key count.
A fixed key-count cap is a poor memory proxy when accumulators vary in size, so workers can OOM well below max_keys (or flush needlessly when accumulators are tiny). Request: bound the table by estimated in-memory size instead of (or in addition to) key count.
Notes:
b/36567833is a Google-internal Buganizer ref with no public page; this issue tracks it publicly.- Sibling
PGBKOperationhas the same shape (max_size = 10 * 1000elements). - Sizing accurately is the hard part —
sys.getsizeofundercounts wrapped/native (e.g. C-extension) accumulators; a coder-based or sampled-RSS estimate is likely needed.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
- Component: Python SDK
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 368
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/beam
-
awaiting triage bug P3
Difficulty 1/5 Under an hour Newbie friendliness 67/100
apache/beam#41197 · 2 comments ·
Maintainers usually reply within 1 day
-
[Bug]: [Python] WriteToJson silently rounds floats to 10 decimal placesPossibly taken @Abhi2939 claimed this 1 day ago. Openio P2 python yaml
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apache/beam#41102 · 4 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/beam#39624 · 2 reactions ·
Maintainers usually reply within 1 day
-
[Bug]: PubsubIO used in batch incorrect batch cutoff sizePossibly taken @1fanwang claimed this 78 days ago. Openbug io P3 pinned pubsub
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/beam#28011 · 4 comments ·
Maintainers usually reply within 1 day
-
build P3 sub-task
Difficulty 1/5 Under an hour Newbie friendliness 68/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
SasanLabs/VulnerableApp#912 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
desktop edt not responding
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
datafusion-contrib/StreamFusion#653 ·
Maintainers usually reply within 1 day
-
[Bug] Absolute row/column index 0 is ignored for images, hyperlinks and commentsPossibly taken @fanOfSunny claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day