Keep exposure stores node-local in single-allocation runs; persist only per-exposure products

Ouverte
  • Quelque chose ne va pas ?
#953 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@cailmdaley y travaille déjà.

Depuis le 10/10/2026.

  • #954 par @cailmdaley — ouverte

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
18/100
Type d'issue
Refactorisation
Clarté
Plutôt claire
Activité
Active
Stack technique
hpc, python
Domaine
data, infrastructure

Piste de recherche

Start with how sp run --in-allocation sets the exposure store location, and find where tile stores already use node-local $SLURM_TMPDIR. Then check sp_check_products.py and the star catalogue merge for any direct read of an exposure store. Done means exposure stores live on node-local disk in allocation mode, only the per-exposure PSF tars and exposure-map fragments are persisted, and a resumed failed batch still completes.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

On Nibi a production batch is now one 144-core allocation running about 25 tiles (sp run --in-allocation). Its tile work is already node-local (#947 and the node-local tile stores). Its exposure stores are not: each batch builds about 43 of them on /scratch. Each node has 3 TB of local disk, so the exposure stores should live there too, and only the per-exposure products the catalogue needs should be persisted.

What the first production wave measured (smk-b01–b05, 10 Oct)

batch tiles exposures wall billed core-h/tile
b01 25 42 2h09m 12.4
b02 25 43 1h58m 11.3
b03 25 42 2h08m 12.3
b04 25 43 2h06m 12.1
b05 25 44 running
  • Scratch bytes are what binds. With wave 1's stores live, /scratch reached ~1.9 TiB against Nibi's 1 TiB per-user soft quota, and it dropped back to 783 GiB as batches finished. The scale is ~6.6 GiB per exposure store, ~300 GiB per batch. Above the soft quota a 60-day grace counter runs, and it resets only when usage drops back below 1 TiB. So on /scratch we can run only 2–3 batches at a time, or keep relying on the grace period.
  • File count does not bind at this scale. It is ~1,000 files per exposure store, ~50K per batch.
  • Boundary recompute is real but cheap. The five batches build 214 exposure stores for 156 distinct exposures, so 27% are recomputed at batch edges. The exposure chain is ~1 of the ~12 billed core-h per tile (g12: 0.99), so the recompute costs about 2% of the total (inferred from g12's split).

Proposal

In --in-allocation mode, put the exposure stores on node-local disk ($SLURM_TMPDIR), next to the tile stores, and persist to the products directory only what downstream reads: the per-exposure PSF tars and the exposure-map fragments.

  • A batch's /scratch footprint goes to roughly its products, so the soft quota stops limiting how many batches run at once. It is then limited by the scheduler and fairshare.
  • It makes #951 (packing per-exposure intermediates for the 1M-file quota) unnecessary for single-allocation runs. #951 still matters for the multi-job Slurm-executor launch.
  • I/O moves off shared NFS. This is the same effect #947 had for tile_detect: 8.7 → 1.0 min.

Not proposed: a cross-batch shared exposure store. It would save the ~2% recompute, but it would bring back a persistent shared store, with locking between concurrent batches and a bigger /scratch footprint. Ordering batches as compact 2D blocks gets most of that 2% for free.

What to check while implementing:

  • Peak node-local use per batch. It should be ~300 GiB stores plus tile stores, well under 3 TB, but measure it on a dense batch.
  • That a failed batch can be resumed. With node-local stores it re-runs from the start: ~2 h, which is acceptable.
  • That nothing downstream (the star catalogue merge, the coverage and defect maps, sp_check_products.py) reads an exposure store directly rather than a persisted product.

Claude (Opus, nibi chair) on behalf of Cail. Measurements by the nibi surveyor session from the smk-b01–b05 ledger.

Langage dominant
Python
Étoiles
18
Forks
14
Merge moyen
3 j 18 h
PR mergées (30 j)
27

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de CosmoStat/shapepipe

Toutes les issues de CosmoStat/shapepipe

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.