a proposal to simplify encoding for zarr data
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start with the zarr-metadata v2 and v3 array.py definitions linked in the issue, then trace xarray’s current encoding handling for zarr.create_array kwargs. Determine how zarr_array_metadata and runtime configuration should coexist with legacy encoding keys and how consistency should be tested. Done means supporting a durable v2/v3 metadata representation without breaking existing encoding behavior.
Written by the indexing model from the issue text.
Description
What is your issue?
When used for zarr data, the .encoding dict is populated with keys derived from the signature of a high-level function (zarr.create_array). That function is abstract, and deliberately elides differences between zarr v2 and zarr v3 data, and obscures important properties of the an array, like the exact chunk grid metadata it uses.
Better than the signature of create_array is an array metadata document itself. The array metadata document is small and defined by the zarr specs, so it's independent of zarr-python changes. You could represent the array metadata as a dict, or as a dataclass, if that's more ergonomic (a dict is simpler in case there are extra fields in metadata).
I think this could work by adding a new zarr_array_metadata key to the encoding dict. The value is a dict that models either a zarr v2 or zarr v3 metadata document. Zarr python (and any other zarr implementation) can create an array from a plain metadata dict. Note: the metadata is a complete description of a stored array, but it doesn't include runtime configuration like write_empty_chunks, so you might need a separate zarr_array_config key for this, if you want to consolidate that kind of configuration info. In any case, the full metadata document is a much more durable representation of an array than what you are using today (a subset of the signature of create_array)
xarray would need a bit of machinery to ensure that the old encoding keys still work, that their value is consistent with the array metadata, and that their use is ultimately deprecated. And it's likely that we need to improve things on the zarr-metadata side, to make the static types better. But I think this is a much better direction than putting create_array kwargs in your encoding dict.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 74
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pydata/xarray
-
bug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
`DataTree.map_over_datasets` behavior can be obscure sometimesPossibly taken @mohansree14 claimed this 23 days ago. Openneeds triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
pydata/xarray#11813 · 1 comment ·
Maintainers usually reply within 1 day
-
Comparing attrs that contain a dict of NumPy arrays raises ValueError in identical / assert_identicalPossibly taken @SohamXYZDev claimed this 29 days ago. Openbug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
pydata/xarray#11777 · 1 comment ·
Maintainers usually reply within 1 day
-
RangeIndex` with datetime64 bounds cannot be printed (`Unknown format code 'g'`)Possibly taken @d-chambers claimed this 60 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pydata/xarray#11598 · 3 comments ·
Maintainers usually reply within 1 day
-
DataTree constructor error message says dict even though dict is not acceptedPossibly taken @aryansk claimed this 100 days ago. Openbug contrib-good-first-issue topic-DataTree topic-error reporting
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pydata/xarray#11514 · 2 comments ·
Maintainers usually reply within 1 day
Similar issues
-
tests
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
[Plugin] 群聊记忆 · 人格养成Open
Difficulty 1/5 1-3 hours Newbie friendliness 74/100
-
bug maintainer-managed
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
awslabs/visual-asset-management-system#1759 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100