`open_grid` on large SCRIP grids OOMs: corner dedup ignores `chunks=`

Open
#1,774 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
numpy, python
Domain
data, performance

Research direction

Read _to_ugrid and the SCRIP reader path first, then trace how chunks= is handled. Done means large SCRIP grids no longer require the full corner table in memory and the 63 GB CONUS-RRM case can open without OOM.

Written by the indexing model from the issue text.

Description

  • SCRIP stores one (lon, lat) per face-corner, so the corner table is n_face × n_corners rows.
  • _to_ugrid dedups it with Polars, which needs the whole table resident: 53.6 GiB of float64 for a 300M-element np4 grid, before the join copy.
  • The reader never dispatches on chunks=, so large SCRIP files OOM no matter how they're opened.
  • This blocks the 63 GB CONUS-RRM np4 grid served through the uxarray-mcp-server, which already passes chunks="auto" and still OOMs on the worker.
Dominant language
Python
Stars
228
Forks
55
Avg merge
4d 12h
Merged PRs (30d)
13

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from UXARRAY/uxarray

All issues in UXARRAY/uxarray

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.