Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Bounding-box subset allocates memory proportional to the whole mesh, not the crop

未关闭
  • 哪里不对？
#1,778 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

@rajeeja 已经在做这个了。

开始于 2026年9月22日。

  • #1779 来自 @rajeeja —— 未关闭

评估

难度
5/5
预计耗时
一周以上
新手友好度
30/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
numpy, python
领域
data, performance

调研方向

Start with uxarray/grid/bounds.py:84-96 and the subset.bounding_box entry point, then run the provided MCVE at both mesh sizes. Review the proposed prefiltering, persisted-connectivity, and file-level SCRIP approaches, including the measurements from #1779. Done means a regional crop no longer incurs memory proportional to the full mesh, with the crop results still matching existing behavior.

由索引模型根据 Issue 内容生成。

描述

scalability
Version

2026.9.0 (and main)

How did you install UXarray?

Source

What happened?

Two separate costs make a regional crop of a large SCRIP mesh scale with the whole mesh rather than the crop. The first is fixable in the subsetting code; the second is a reader/representation question and is the harder one.

1. Grid.bounds is computed for every face before any filtering.

subset.bounding_box filters on face_bounds_lon/lat, which come from _populate_face_bounds (uxarray/grid/bounds.py:84-96). That materializes the node coordinate and connectivity arrays with .values regardless of how the grid was opened. Measured, 12-corner SCRIP, crop fixed at ~0.006% of the mesh:

n_face crop memory
400,000 26 faces +0.076 GiB
1,600,000 116 faces +0.851 GiB

Asking for a smaller box does not help — the index is built before the box is consulted.

2. face_node_connectivity is a lazy graph over the dedup, so any slice replays it.

On the 63 GB CONUS-RRM np4 grid (299,999,162 faces), after open_grid leaves ~119 GiB resident, even a contiguous conn[:30M] kills a 236 GiB worker: computing the connectivity re-reads the 53.6 GiB corner arrays.

The decisive measurement is crop-size independence:

crop candidates outcome
Texas 30,914,134 killed on slice
Key West 15,866 killed on slice

A 2000× smaller crop fails identically, so the cost is not the crop.

For contrast, the pg2 twin — same 200 m geometry, 4 corners instead of 12 — subsets in 113 s.

What did you expect to happen?

That a regional crop costs something proportional to the region, and that a mesh which can be opened can also be subset.

Can you provide a MCVE to reproduce the bug?
"""Memory cost of a bbox subset scales with the mesh, not the crop."""
import gc, os, sys, tempfile, time
from pathlib import Path
import numpy as np, psutil, uxarray as ux, xarray as xr

P = psutil.Process(os.getpid())
rss = lambda: P.memory_info().rss / 1024**3
n_face, n_corner = int(sys.argv[1]), 12

tmp = Path(tempfile.mkdtemp()) / "mesh.nc"
rng = np.random.default_rng(0)
clon, clat = rng.uniform(0, 360, n_face), rng.uniform(-85, 85, n_face)
off = np.linspace(0, 2 * np.pi, n_corner, endpoint=False)
xr.Dataset({
    "grid_corner_lon": (("grid_size", "grid_corners"),
                        (clon[:, None] + 0.02 * np.cos(off)[None, :]) % 360.0),
    "grid_corner_lat": (("grid_size", "grid_corners"),
                        np.clip(clat[:, None] + 0.02 * np.sin(off)[None, :], -90, 90)),
    "grid_center_lon": (("grid_size",), clon),
    "grid_center_lat": (("grid_size",), clat),
    "grid_imask": (("grid_size",), np.ones(n_face, "i4")),
    "grid_area": (("grid_size",), np.zeros(n_face)),
}).to_netcdf(tmp)
del clon, clat; gc.collect()

grid = ux.open_grid(str(tmp), chunks="auto")
base = rss(); t0 = time.perf_counter()
sub = grid.subset.bounding_box(lon_bounds=[-1.0, 1.0], lat_bounds=[-1.0, 1.0])
print(f"n_face={n_face:,} crop={int(sub.n_face):,} +{rss()-base:.3f} GiB "
      f"{time.perf_counter()-t0:.1f}s")

Run at 400000 and 1600000: per-face cost stays flat while the crop stays at ~0.006%.

Possible directions

Not prescribing a fix; each is a design call.

  1. Prefilter on face centres before computing exact bounds — addresses (1). #1779 does this and measures 0.84 GiB → 0.00 GiB at 1.6M faces.
  2. Persist connectivity after dedup, so slicing does not replay the graph — addresses (2).
  3. Subset SCRIP at the file level, bypassing global connectivity entirely. grid_center_lat/lon gives candidates and grid_corner_lat/lon settles them exactly; neither needs node dedup. Prototyped outside uxarray against the 63 GB file: Key West returns 8,858 faces in 39 s at 0.47 GiB, verified to return the identical face set to subset.bounding_box on every SCRIP mesh small enough for both to run. The file is chunking=contiguous, so candidate rows are direct seeks.
  4. Do not hold both coordinate representations: node_x/y/z are derivable from node_lon/lat and are ~46% of what survives a subset.

A narrower INT_DTYPE for connectivity would halve another array, but #1658 shows INT_FILL_VALUE does not survive that narrowing — mentioned only to rule it out.

主要语言
Python
星标
230
派生
56
平均合并
6 天 3 小时
30 天内合并 PR
12

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

UXARRAY/uxarray 的其他 Issue

查看 UXARRAY/uxarray 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。