Simultaneously read multiple Datasets into an Xarray-Beam pipeline
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
Research direction
Start by reading the existing xbeam.DatasetToChunks entry point and the current beam.CoGroupBykey approach described in the issue. Define how a single transform should accept multiple xarray.Dataset objects and emit keyed dataset tuples without the extra disk write. Done means the proposed tuple output works for multiple datasets in an Xarray-Beam pipeline.
Written by the indexing model from the issue text.
Description
It is relatively common to need to load multiple xarray.Dataset objects, e.g., to compare two different models.
This currently can be done by loading data with separate calls to xbeam.DatasetToChunks, and by joining together the result with beam.CoGroupBykey. This works but is rather inefficient, involving an extra write of the data to disk. Ideally we could load the data in a single beam transform instead, e.g., xbeam.DatasetToChunks([ds1, ds2], chunks) would return a PCollection with elements of type tuple[xbeam.Key, tuple[xarray.Dataset, xarray.Dataset]].
CC @alxmrs
- Dominant language
- Python
- Stars
- 170
- Forks
- 15
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from google/xarray-beam
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
google/xarray-beam#196 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
google/xarray-beam#199 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
google/xarray-beam#195 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
google/xarray-beam#173 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
google/xarray-beam#124 ·
All issues in google/xarray-beam
Similar issues
-
bug confirmed issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
open-webui/open-webui#30750 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
OpenwaterHealth/openmotion-bloodflow-app#604 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
good first issue
Difficulty 1/5 Under an hour Newbie friendliness 90/100