Run 2: publish the survey as order-5 HATS pixels, uploaded once per pixel, replacing latest/ and snapshots

Ouverte
  • Quelque chose ne va pas ?
#987 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@cailmdaley y travaille déjà.

Depuis le 30/10/2026.

  • #992 par @cailmdaley — ouverte

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
25/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
python
Domaine
data-engineering, distributed-systems

Piste de recherche

Read the run-2 publishing proposal and the linked #986 tile planning issue first; the payload names no implementation files or tests. Trace the existing latest/ merge, snapshot upload, and publisher proposed in #982, then check the prototype path and normal launcher on Nibi and Fir. Done means meeting the acceptance checks: publish seam-crossing pixels once, match source rows and maps, refuse duplicate uploads, and support authenticated lsdb.read_hats cone searches while pixels are added.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Proposal for run 2: replace the latest/ merge and the VOSpace snapshots with per-pixel HATS publication. Each cluster builds its order-5 pixels from finished tiles and uploads each one once, and the catalogue's pixel list is rebuilt from the VOSpace listing.

This is the publishing half of the run-2 design; #986 covers the tile list, ownership and planning. If adopted, it replaces the merged HDF5 catalogue, its 8–20 h rebuilds, the snapshot uploads and the survey-wide publisher that #982 proposes to port.

Building a pixel. A pixel is built when every tile and exposure on its fixed list (#986) has finished. A tile that failed after its retries contributes no rows, as a failed unit is kept out of the catalogue today. The builder reads the finished tile catalogues on the list, keeps the rows whose own position (_healpix_29 >> 48) falls in the pixel, and writes one Parquet file. Rows are placed by their own position, never by their tile's centre: placing whole tiles would put 13.6% of rows outside their pixel, and cone searches would silently miss them. Tile overlaps are kept, as in run 1.

Products, under vos:cfis/weak_lensing/ShapePipe/v2.0/:

  • the galaxy catalogue and the PSF-validation star catalogue, as HATS;
  • the coverage maps npoint, ntaken, nexp and nflagged (order 17), as HEALSparse, one file per order-5 pixel.

Each file records the ShapePipe commit that made it, and the path and checksum of the UNIONS mask it was built against, in its metadata. The mask itself is independent of ShapePipe and is not copied. PSF model tarballs and tile working directories stay on the clusters.

Upload. The owning cluster uploads each pixel once and checks size and MD5 against CADC; an uploaded pixel is never overwritten. properties (without row counts) and dataset/_common_metadata are written once. partition_info.csv is the only file that changes: a cluster rebuilds it from the VOSpace listing after an upload, so it never names a missing file, and if two clusters race, the next rebuild corrects it. _metadata is omitted. lsdb 0.11.1 reads such a catalogue correctly; len(cat) is unknown while the catalogue grows.

No versioning layer. The catalogue is v2.0/, and run 1 is converted into it. Keeping the code consistent across the two clusters is the operator's job, and the commit recorded in each file shows any mismatch.

Acceptance.

  • A small two-cluster patch with the seam inside it publishes every pixel once, run through the normal launcher on Nibi and Fir.
  • Its pixels match the source tile rows value for value, and its maps match the run-1 merged maps on the same patch.
  • A second upload of a published pixel is refused.
  • lsdb.read_hats and a cone search work over authenticated HTTPS while pixels are being added.
  • Margin catalogues (for crossmatch) are built once a pixel's neighbours are published, outside this path.

Prototype. Four pixels were published to vos:cfis/weak_lensing/ShapePipe/v2.x/run2-proto-hats/ and read back with lsdb over HTTPS, with a 6.5 s cone search. Every value of all 191 galaxy and 22 star columns matched latest/. The area is not public, so readers need a CADC proxy certificate. CADC returns the wrong bytes for suffix byte ranges (bytes=-8); fsspec uses explicit ranges and is not affected.

— Claude (Opus 5.5) on behalf of Cail

Langage dominant
Python
Étoiles
18
Forks
14
Merge moyen
3 j 18 h
PR mergées (30 j)
27

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de CosmoStat/shapepipe

Toutes les issues de CosmoStat/shapepipe

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.