Some notes on alignment with pangeo-forge-recipes around "keys"
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Refactor
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
Research direction
Start by reading the Xarray Beam keys documentation and the referenced pangeo_forge_recipes/patterns.py lines 70-114. Compare those abstractions with the project's handling of indexes in FilePatterns and parallel Zarr writing. The issue does not define a concrete change or acceptance criteria, so a maintainer decision is needed before implementation.
Written by the indexing model from the issue text.
Description
At today's Pangeo Forge meeting, @alxmrs told us a bit more about Xarray Beam. I'm happy to note that we are aligning a bit around certain abstractions. I'm not proposing to merge or share code or anything at this stage. Just happy to see that we have independently arrived at some similar concepts. Comparing side by side may also suggest improvements.
Xarray Beam
https://xarray-beam.readthedocs.io/en/latest/data-model.html#keys-in-xarray-beam
To keep track of how individual records could be combined into a larger (virtual) dataset, Xarray-Beam defines a Key object. > Key objects consist of:
- offsets: integer offests for chunks from the origin in an immutabledict
- vars: The subset of variables included in each chunk, either as a frozenset, or as None to indicate “all variables”.
Pangeo Forge
We currently call this an "index," although I agree that key is a better name. Our use of indexes is less well documented as it is not really public API but more of an internal thing. We mention it here: https://pangeo-forge.readthedocs.io/en/latest/recipe_user_guide/file_patterns.html#inspect-a-filepattern
The index is its own special type of object used internally by recipes, a
pangeo_forge_recipes.patterns.Index, (which is basically a tuple of one or morepangeo_forge_recipes.patterns.DimIndexobjects).
The code is pretty straightforward and the comments explain things better.
These indexes are used both by FilePatterns (see #31) as well as for parallel writing of Zarr datasets.
cc @cisaacstern
- Dominant language
- Python
- Stars
- 170
- Forks
- 15
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from google/xarray-beam
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
google/xarray-beam#196 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
google/xarray-beam#199 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
google/xarray-beam#195 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
google/xarray-beam#173 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
google/xarray-beam#124 ·
All issues in google/xarray-beam
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
anthropics/skills#1811 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
speaches-ai/speaches#678 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
datalayer/mcp-compose#42 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
conda-forge/spacy-feedstock#177 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
UKGovernmentBEIS/inspect_evals#2523 ·