Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Keep exposure stores node-local in single-allocation runs; persist only per-exposure products

Aperta
  • Qualcosa non va?
#953 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@cailmdaley ci sta già lavorando.

Dal 10/10/2026.

  • #954 di @cailmdaley — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
18/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
hpc, python
Ambito
data, infrastructure

Direzione di ricerca

Start with how sp run --in-allocation sets the exposure store location, and find where tile stores already use node-local $SLURM_TMPDIR. Then check sp_check_products.py and the star catalogue merge for any direct read of an exposure store. Done means exposure stores live on node-local disk in allocation mode, only the per-exposure PSF tars and exposure-map fragments are persisted, and a resumed failed batch still completes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

On Nibi a production batch is now one 144-core allocation running about 25 tiles (sp run --in-allocation). Its tile work is already node-local (#947 and the node-local tile stores). Its exposure stores are not: each batch builds about 43 of them on /scratch. Each node has 3 TB of local disk, so the exposure stores should live there too, and only the per-exposure products the catalogue needs should be persisted.

What the first production wave measured (smk-b01–b05, 10 Oct)

batch tiles exposures wall billed core-h/tile
b01 25 42 2h09m 12.4
b02 25 43 1h58m 11.3
b03 25 42 2h08m 12.3
b04 25 43 2h06m 12.1
b05 25 44 running
  • Scratch bytes are what binds. With wave 1's stores live, /scratch reached ~1.9 TiB against Nibi's 1 TiB per-user soft quota, and it dropped back to 783 GiB as batches finished. The scale is ~6.6 GiB per exposure store, ~300 GiB per batch. Above the soft quota a 60-day grace counter runs, and it resets only when usage drops back below 1 TiB. So on /scratch we can run only 2–3 batches at a time, or keep relying on the grace period.
  • File count does not bind at this scale. It is ~1,000 files per exposure store, ~50K per batch.
  • Boundary recompute is real but cheap. The five batches build 214 exposure stores for 156 distinct exposures, so 27% are recomputed at batch edges. The exposure chain is ~1 of the ~12 billed core-h per tile (g12: 0.99), so the recompute costs about 2% of the total (inferred from g12's split).

Proposal

In --in-allocation mode, put the exposure stores on node-local disk ($SLURM_TMPDIR), next to the tile stores, and persist to the products directory only what downstream reads: the per-exposure PSF tars and the exposure-map fragments.

  • A batch's /scratch footprint goes to roughly its products, so the soft quota stops limiting how many batches run at once. It is then limited by the scheduler and fairshare.
  • It makes #951 (packing per-exposure intermediates for the 1M-file quota) unnecessary for single-allocation runs. #951 still matters for the multi-job Slurm-executor launch.
  • I/O moves off shared NFS. This is the same effect #947 had for tile_detect: 8.7 → 1.0 min.

Not proposed: a cross-batch shared exposure store. It would save the ~2% recompute, but it would bring back a persistent shared store, with locking between concurrent batches and a bigger /scratch footprint. Ordering batches as compact 2D blocks gets most of that 2% for free.

What to check while implementing:

  • Peak node-local use per batch. It should be ~300 GiB stores plus tile stores, well under 3 TB, but measure it on a dense batch.
  • That a failed batch can be resumed. With node-local stores it re-runs from the start: ~2 h, which is acceptable.
  • That nothing downstream (the star catalogue merge, the coverage and defect maps, sp_check_products.py) reads an exposure store directly rather than a persisted product.

Claude (Opus, nibi chair) on behalf of Cail. Measurements by the nibi surveyor session from the smk-b01–b05 ledger.

Lingua principale
Python
Stelle
18
Fork
14
Merge medio
2g 13h
PR unite (30g)
35

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di CosmoStat/shapepipe

Tutte le issue di CosmoStat/shapepipe

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.