Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

One ShapePipe workflow for data and image sims

Aperta
#891 1 commento 0 reazioni 2 assegnatari Vedi su GitHub

@martinkilbinger ci sta già lavorando.

Dal 12/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

The image-sims ShapePipe run is a separate orchestrator (example/cfis_image_sims plus the *_canfar_v2.0.bash dispatcher, driven from sp_validation's image_sims.smk), and the config cleanup that came with the data workflow (#847, #852) has broken it: its default.* symlinks dangle and it selects mask configs that no longer exist. Meanwhile the Snakemake workflow (#852) does everything the sims need — chunked ngmix, node-local vignette staging, completeness checks, resume, the run report — and the sims get none of it.

Target: sims are a branch of the one workflow, not a sibling.

  • rules/image_sims.smk in the main tree. Two config keys select the branch: kind: data|sims (where the pixels come from and how they're named) and psf_model: psfex|mccd|fake. They stay separate so sims can also run the real PSF chain once the star-injected grids exist.
  • The sims delta is small: a few _sims ini variants under workflow/config/cfis/, one new rule (tile_fake_psf), and tile_uncompress/exp_psf skipped for sims. Everything else is the data DAG unchanged.
  • Each shear branch (1z2z, 1p2z, …) is one campaign: five sp runs over the same sky with different input roots. sp_validation's sims workflow drives ShapePipe — its m-bias DAG calls sp run per branch, then reads only the campaign products (#879's final_cat_<campaign>.hdf5 plus a run manifest), never ShapePipe's run tree.
  • The workflow runs on both nibi and candide.
  • Gone: example/cfis_image_sims, the bash dispatcher (after a deprecation release — they're packaged entry points), and sp_validation's im_init/im_pipeline/im_merge with their ShapePipe-internal config keys.

Steps — each becomes a sub-issue when we're ready to pick it up:

  1. sp run --config-file and a candide profile (shapepipe). No dependencies; this is the two-machine work.
  2. #879 (in progress). The sims branch stacks on its final_cat_merge.
  3. The sims branch (shapepipe): kind/psf_model keys, _sims ini variants, tile_fake_psf, the two elided rules. Acceptance: one tile of grid_3 through sp run, compared against the existing catalogue.
  4. The boundary (sp_validation): run manifest, im_shapepipe calling sp run per branch, cat_config reading the manifest; im_init/im_pipeline/im_merge and their keys removed. The comprehensive catalogue comes from the v2 hdf5 readers (CosmoStat/sp_validation#343) rather than extract_info.py (CosmoStat/sp_validation#294).
  5. Deprecate example/cfis_image_sims and the bash dispatcher (shapepipe), one release later removal.

Supersedes the pipeline half of CosmoStat/sp_validation#284 / #285 / #287; related: #870, #885.

Rule table (data vs sims)
Rule data sims Parameterisation
tile_get_images ✓ ✓ config_tile_Git_{kind}.ini (INPUT_PATH, INPUT_FILE_PATTERN, INPUT_NUMBERING \d{3}\.\d{3} vs \d{3}-\d{3}, INPUT_FILE_EXT, OUTPUT_FILE_PATTERN). Plus a rule-side change: get_images_runner matches tile_numbers.txt, which unit_pre writes in dot format; sims need the dashed id, so unit_pre branches on kind (or exports a third var). Not ini-only.
tile_uncompress ✓ — Elided for sims (weights ship uncompressed); sims' Sx reads the weight from Git directly. Works because A's inis use fixed absolute INPUT_DIRs. Kills bit 2.
tile_find_exposures ✓ ✓ config_tile_Fe_{kind}.ini — not one ini. Verified diff: FILE_PATTERN CFIS_image→CFIS_simu_image, COLNUM 3→2, EXP_PREFIX p→simu_image-. Those two are FITS-HISTORY parsing facts, they define the tile↔exposure edge build_index.py consumes, and getting them wrong yields a wrong or empty exposure list silently.
exp_get_images ✓ ✓ config_exp_Gie_{kind}.ini (patterns + roots).
exp_split ✓ ✓ One ini; FILE_EXT → $SP_IMG_EXT. N_HDU = 40 identical.
exp_psf ✓ only if psf_model != fake Unchanged for data (post-#888). A sims × psfex variant is a fourth ini — see §2.1.
tile_fake_psf (new) — only if psf_model == fake Tile-level, between tile_detect and tile_vignets. Must join the tile_shape group — fake_psf_runner writes galaxy_psf*.sqlite, the store ngmix reads at random per object per epoch, which is exactly what node-local NVMe staging exists for; outside the group it lands on NFS. Joining means inheriting the group's resource contract, and NGMIX_VIGNET_DIR hardcodes run_sp_tile_PiViVi, so any sims PiViVi variant keeps that RUN_NAME.
tile_exp_forest, tile_merge_headers ✓ ✓ Unchanged (config_tile_Mh_exp_{kind}.ini for WARN_MISSING_EXP/INPUT_DIR).
tile_detect ✓ ✓ config_tile_Sx_{kind}.ini (weight source, image pattern).
tile_vignets ✓ ✓ config_tile_PiViVi_{psf_model}.ini gains a fake variant (vignetmakers only).
tile_ngmix ✓ ✓ Both keys: PSF-source dir on psf_model, background inputs + BKG_SUB on kind. Chunking, ngmix_ranges.json, the tile_shape group apply unchanged — sims gain parallel ngmix, their compute bottleneck, for free.
tile_merge_cats, tile_make_cat ✓ ✓ config_tile_Mc.ini carries no psf suffix on the data side; sims' config_tile_Mc_psfex.ini folds into it.
clean_exposure, clean_tile ✓ ✓ Unchanged.
final_cat_merge (#879) ✓ ✓ Already the campaign-merge rule; sims inherit it.

— Fable on behalf of Cail

Lingua principale
Python
Stelle
18
Fork
14
Merge medio
8h 40m
PR unite (30g)
10

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di CosmoStat/shapepipe

Tutte le issue di CosmoStat/shapepipe

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.