`OverflowError` when a chunk exceeds 2 GB
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
Research direction
Start in xarray_beam/_src/core.py at DatasetCoder.estimate_size around line 275, then reproduce the failure with the minimal Apache Beam pipeline shown in the issue. The work is complete when oversized estimates no longer raise OverflowError and the condition remains visible through a warning.
Written by the indexing model from the issue text.
Description
DatasetCoder.estimate_size returns value.nbytes directly (core.py#L275). For
chunks larger than 2**31 - 1 bytes (~2 GB), this overflows the C int in Beam's Cython CallbackCoderImpl.estimate_size and raises:
OverflowError: value too large to convert to int
Noticed this hitting after upgrading to the latest version when loading datasets from GRIBs with multple atmostpheric levels, e.g., ECMWF IFS ensembles (51 members × 13 levels × 721 × 1440), where a single per-key xarray.Dataset chunk exceeds 2 GB before any rechunk.
Minimal repro:
import apache_beam as beam
import numpy as np
import xarray as xr
import xarray_beam as xbeam
ds = xr.Dataset({"x": (("a", "b"), np.zeros((20000, 14000)))}) # ~2.1 GB
with beam.Pipeline() as p:
_ = (
p
| beam.Create([(xbeam.Key({}), ds)])
| beam.Reshuffle()
)
Capping the returned estimate at 2**31 - 1 (with a warning so oversized chunks are still visible) avoids the crash, will put up a PR.
- Dominant language
- Python
- Stars
- 170
- Forks
- 15
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from google/xarray-beam
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
google/xarray-beam#199 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
google/xarray-beam#195 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
google/xarray-beam#173 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
google/xarray-beam#124 ·
-
Use xbeam to append? Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
google/xarray-beam#121 · 3 comments ·
All issues in google/xarray-beam
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100