Better documentation of behaviour with irregular source chunks
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Documentation
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering
Research direction
Start by reviewing the behavior of Rechunk and ConsolidateChunks with irregular source chunk sizes, including the leap-year example in the issue. Determine whether the limitation is fundamental and what guidance, such as preprocessing with SplitChunks, should be documented; done means the limitation and recommendations are clearly explained.
Written by the indexing model from the issue text.
Description
Hiya!
In some source datasets the original chunks have irregular sizes. For example one netcdf file per year, where some years are leap years hence shorter than others.
Although the data model would seem to support this in principle, I couldn't get Rechunk or ConsolidateChunks to work without regular source chunk sizes.
Is this a fundamental limitation or something that could be in scope to address? And do you have any recommendations for getting data like this into regular chunks? In the leap year case, first splitting the source chunks into one chunk per day using SplitChunks could be an option. Although in general this would require splitting into chunks of gcd(*all_possible_source_chunk_sizes) which could be too small.
- Dominant language
- Python
- Stars
- 170
- Forks
- 15
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from google/xarray-beam
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
google/xarray-beam#196 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
google/xarray-beam#199 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
google/xarray-beam#195 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
google/xarray-beam#173 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
google/xarray-beam#124 ·
All issues in google/xarray-beam
Similar issues
-
bug confirmed issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
open-webui/open-webui#30750 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
OpenwaterHealth/openmotion-bloodflow-app#604 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
good first issue
Difficulty 1/5 Under an hour Newbie friendliness 90/100