Dimension order produced by Rechunk is opaque and not controllable; mismatch can cause errors from subsequent ChunksToZarr.
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
Research direction
Start by tracing the Rechunk-to-ChunksToZarr pipeline described in the report and reproduce the shown dimension-order mismatch. Compare the dimensions emitted by Rechunk with those expected by the ChunksToZarr template; done means the ordering can be controlled or equivalent dimensions are safely handled before writing.
Written by the indexing model from the issue text.
Description
In a pipeline in which Rechunk is followed by ChunksToZarr, one can run into errors when the dimension order of variables output by Rechunk doesn't match that of the template you pass to ChunksToZarr, resulting in errors like:
ValueError: variable 'geopotential_quantiles' already exists with different dimension names ('hour', 'dayofyear', 'level', 'latitude', 'longitude', 'quantile') != ('level', 'hour', 'dayofyear', 'latitude', 'longitude', 'quantile'), but changing variable dimensions is not supported by to_zarr().
As far as I can tell Rechunk doesn't allow you to control the output dimension order (at least, not on a per-variable basis, which may be necessary to match a given template). An alternative could be to transpose the output template instead to match whatever Rechunk is going to produce, but it's hard to know what that's going to be as well.
As another way around this, it'd be nice if ChunksToZarr could just do the transpose rather than complain if it finds this kind of dimension mismatch (same dimensions in a different order).
- Dominant language
- Python
- Stars
- 170
- Forks
- 15
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from google/xarray-beam
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
google/xarray-beam#196 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
google/xarray-beam#199 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
google/xarray-beam#195 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
google/xarray-beam#173 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
google/xarray-beam#124 ·
All issues in google/xarray-beam
Similar issues
-
bug confirmed issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
open-webui/open-webui#30750 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
OpenwaterHealth/openmotion-bloodflow-app#604 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
good first issue
Difficulty 1/5 Under an hour Newbie friendliness 90/100