Codec-based alternative to attribute-based scale-offset encoding

Open
#11,280 6 comments 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
numpy, python
Domain
data

Research direction

Start by reading the scale-offset encoding logic linked in xarray/coding/variables.py, then review the Zarr V3 cast_value and scale_offset codec specifications and the demonstration gist. The issue does not define a concrete integration path or acceptance criteria, so done would first require agreeing on the approach before implementation.

Written by the indexing model from the issue text.

Description

enhancement topic-metadata topic-zarr
What is your issue?

Zarr V3 has 2 new codec specs that might be of interest to xarray. We have a codec for casting arrays from one data type to another (cast_value) and a codec for applying a scale + offset transformation to an array (scale_offset). Together, these codecs can define the scale-offset encoding commonly used for compressing floating point measurements as ints before serialization, and which is currently implemented here in this library. The cast_value codec also supports remapping scalars within the same data type (e.g., mapping NaN to 0, and the reverse), which might also be useful in xarray.

see this gist for a demo. it just shows that the combination of a scale_offset codec (defined inline) and the cast_value codec gets the same results as the cf-style zarr attributes + xarray's encoding / decoding logic.

the advantage of pushing this logic down into Zarr itself is that it makes the encoding portable across zarr implementations, and it takes load off the zarr array attributes, which IMO are not well suited for declaring how array scalars should be encoded / decoded.

a disadvantage is that you need implementations of 2 new codecs: I cooked up an implementation of the cast_value codec in the cast-value.py package. There's a default numpy implementation and an optional rust implementation. The rust version has generally better performance from a CPU and memory perspective. I opened a PR in zarr-python to add the numpy cast_value implementation, and a scale_offset codec (which is dead simple) but it didn't get any traction so I closed it 😆

I don't know what integration path makes sense in xarray, so I can't propose any concrete code changes! But it might be worth thinking about how you can make codecs do more, and attributes do less, for data written to zarr.

Dominant language
Python
Stars
4.2k
Forks
1.5k
Avg merge
2d 21h
Merged PRs (30d)
74

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pydata/xarray

All issues in pydata/xarray

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.