Codec-based alternative to attribute-based scale-offset encoding
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start by reading the scale-offset encoding logic linked in xarray/coding/variables.py, then review the Zarr V3 cast_value and scale_offset codec specifications and the demonstration gist. The issue does not define a concrete integration path or acceptance criteria, so done would first require agreeing on the approach before implementation.
Written by the indexing model from the issue text.
Description
What is your issue?
Zarr V3 has 2 new codec specs that might be of interest to xarray. We have a codec for casting arrays from one data type to another (cast_value) and a codec for applying a scale + offset transformation to an array (scale_offset). Together, these codecs can define the scale-offset encoding commonly used for compressing floating point measurements as ints before serialization, and which is currently implemented here in this library. The cast_value codec also supports remapping scalars within the same data type (e.g., mapping NaN to 0, and the reverse), which might also be useful in xarray.
see this gist for a demo. it just shows that the combination of a scale_offset codec (defined inline) and the cast_value codec gets the same results as the cf-style zarr attributes + xarray's encoding / decoding logic.
the advantage of pushing this logic down into Zarr itself is that it makes the encoding portable across zarr implementations, and it takes load off the zarr array attributes, which IMO are not well suited for declaring how array scalars should be encoded / decoded.
a disadvantage is that you need implementations of 2 new codecs: I cooked up an implementation of the cast_value codec in the cast-value.py package. There's a default numpy implementation and an optional rust implementation. The rust version has generally better performance from a CPU and memory perspective. I opened a PR in zarr-python to add the numpy cast_value implementation, and a scale_offset codec (which is dead simple) but it didn't get any traction so I closed it 😆
I don't know what integration path makes sense in xarray, so I can't propose any concrete code changes! But it might be worth thinking about how you can make codecs do more, and attributes do less, for data written to zarr.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 74
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pydata/xarray
-
bug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
`DataTree.map_over_datasets` behavior can be obscure sometimesPossibly taken @mohansree14 claimed this 23 days ago. Openneeds triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
pydata/xarray#11813 · 1 comment ·
Maintainers usually reply within 1 day
-
Comparing attrs that contain a dict of NumPy arrays raises ValueError in identical / assert_identicalPossibly taken @SohamXYZDev claimed this 29 days ago. Openbug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
pydata/xarray#11777 · 1 comment ·
Maintainers usually reply within 1 day
-
RangeIndex` with datetime64 bounds cannot be printed (`Unknown format code 'g'`)Possibly taken @d-chambers claimed this 60 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pydata/xarray#11598 · 3 comments ·
Maintainers usually reply within 1 day
-
DataTree constructor error message says dict even though dict is not acceptedPossibly taken @aryansk claimed this 100 days ago. Openbug contrib-good-first-issue topic-DataTree topic-error reporting
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pydata/xarray#11514 · 2 comments ·
Maintainers usually reply within 1 day
Similar issues
-
tests
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
[Plugin] 群聊记忆 · 人格养成Open
Difficulty 1/5 1-3 hours Newbie friendliness 74/100
-
bug maintainer-managed
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
awslabs/visual-asset-management-system#1759 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100