multiple reads of nasa-nex-gddp-cmip6 dataset from MultiZarrToZarr concatenated metadata returns all nans

Open
#289 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
jupyter-notebook
Domain
data-engineering

Research direction

Start with cells 13 and 14 in datasets/nasa-nex-gddp-cmip6/nasa-nex-gddp-cmip6-example.ipynb and reproduce the point time-series read twice. Inspect the MultiZarrToZarr concatenated metadata behavior, including repeated or parallel reads. Done means the second read returns the same valid values as the first rather than all NaNs.

Written by the indexing model from the issue text.

Description

Simply running twice cells 13 & 14 of the notebook that read and plot point single variable time-series for a point will reproduce this issue where the first run will have the valid values but second will be all nans. I encountered this when parallelizing reading of the files with dask that results in multiple reads and the unexpected result.

Dominant language
Jupyter Notebook
Stars
455
Forks
225
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PlanetaryComputerExamples

All issues in microsoft/PlanetaryComputerExamples

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.