Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Keep exposure stores node-local in single-allocation runs; persist only per-exposure products

Abierto
  • ¿Algo no está bien?
#953 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@cailmdaley ya está trabajando en esto.

Desde el 10/10/2026.

  • #954 de @cailmdaley — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
18/100
Tipo de issue
Refactorización
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
hpc, python
Área
data, infrastructure

Línea de trabajo

Start with how sp run --in-allocation sets the exposure store location, and find where tile stores already use node-local $SLURM_TMPDIR. Then check sp_check_products.py and the star catalogue merge for any direct read of an exposure store. Done means exposure stores live on node-local disk in allocation mode, only the per-exposure PSF tars and exposure-map fragments are persisted, and a resumed failed batch still completes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

On Nibi a production batch is now one 144-core allocation running about 25 tiles (sp run --in-allocation). Its tile work is already node-local (#947 and the node-local tile stores). Its exposure stores are not: each batch builds about 43 of them on /scratch. Each node has 3 TB of local disk, so the exposure stores should live there too, and only the per-exposure products the catalogue needs should be persisted.

What the first production wave measured (smk-b01–b05, 10 Oct)

batch tiles exposures wall billed core-h/tile
b01 25 42 2h09m 12.4
b02 25 43 1h58m 11.3
b03 25 42 2h08m 12.3
b04 25 43 2h06m 12.1
b05 25 44 running
  • Scratch bytes are what binds. With wave 1's stores live, /scratch reached ~1.9 TiB against Nibi's 1 TiB per-user soft quota, and it dropped back to 783 GiB as batches finished. The scale is ~6.6 GiB per exposure store, ~300 GiB per batch. Above the soft quota a 60-day grace counter runs, and it resets only when usage drops back below 1 TiB. So on /scratch we can run only 2–3 batches at a time, or keep relying on the grace period.
  • File count does not bind at this scale. It is ~1,000 files per exposure store, ~50K per batch.
  • Boundary recompute is real but cheap. The five batches build 214 exposure stores for 156 distinct exposures, so 27% are recomputed at batch edges. The exposure chain is ~1 of the ~12 billed core-h per tile (g12: 0.99), so the recompute costs about 2% of the total (inferred from g12's split).

Proposal

In --in-allocation mode, put the exposure stores on node-local disk ($SLURM_TMPDIR), next to the tile stores, and persist to the products directory only what downstream reads: the per-exposure PSF tars and the exposure-map fragments.

  • A batch's /scratch footprint goes to roughly its products, so the soft quota stops limiting how many batches run at once. It is then limited by the scheduler and fairshare.
  • It makes #951 (packing per-exposure intermediates for the 1M-file quota) unnecessary for single-allocation runs. #951 still matters for the multi-job Slurm-executor launch.
  • I/O moves off shared NFS. This is the same effect #947 had for tile_detect: 8.7 → 1.0 min.

Not proposed: a cross-batch shared exposure store. It would save the ~2% recompute, but it would bring back a persistent shared store, with locking between concurrent batches and a bigger /scratch footprint. Ordering batches as compact 2D blocks gets most of that 2% for free.

What to check while implementing:

  • Peak node-local use per batch. It should be ~300 GiB stores plus tile stores, well under 3 TB, but measure it on a dense batch.
  • That a failed batch can be resumed. With node-local stores it re-runs from the start: ~2 h, which is acceptable.
  • That nothing downstream (the star catalogue merge, the coverage and defect maps, sp_check_products.py) reads an exposure store directly rather than a persisted product.

Claude (Opus, nibi chair) on behalf of Cail. Measurements by the nibi surveyor session from the smk-b01–b05 ledger.

Lenguaje dominante
Python
Estrellas
18
Forks
14
Merge medio
2 d 13 h
PR fusionados (30 d)
35

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de CosmoStat/shapepipe

Todos los issues de CosmoStat/shapepipe

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.