Persisted outputs cannot be read after a spec version bump

Aperta
  • Qualcosa non va?
#177 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start by reading the workspace output lookup and Store.read behavior described in the issue, then inspect the four options and their tradeoffs. The issue identifies no source files or tests; first locate the workspace and folder store implementations. Done means choosing and implementing a way to read persisted outputs across spec versions or workspaces, with tests that cover the selected behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Since #174, Store.read(record, output, tp) takes the type that the spec declares for the output, and the workspace passes it from the binding of the record's spec:

spec_id = self._views.records.spec(ref.record)       # SpecId(name, version)
binding = self._bindings.get(spec_id)                # None -> NotReadable
tp = binding.spec.outputs.model_fields[ref.output].annotation
value = self._store.read(ref.record, ref.output, tp)

The reason: the folder store writes scipp HDF5, which holds no Python types, so a group (a pydantic model) can only be rebuilt from the declared type.

Consequence: a workspace reads the persisted outputs only of the specs it binds, by name and version.

bindings = [IOFQ v1]   -> persists record r1
upgrade: bindings = [IOFQ v2]
client.output(r1)      -> NotReadable: the workspace does not run loki-iofq v1

So after a version bump, the outputs persisted under the old version cannot be read, although the files are intact, and a request cannot reference them. The same holds for a second workspace that shares the store but runs other specs, which contradicts "workspaces share values only through the store" unless both bind the spec.

Options:

  1. Keep old versions bound. The deployment lists the bindings of every version whose outputs it still wants to read. No code change; the cost moves to whoever maintains the list.
  2. Bind specs for reading without a workflow. A workspace takes specs it only reads (WorkflowSpec without a binding). Reading then needs the spec's package, not its workflow's dependencies.
  3. Store the type with the value, e.g. the qualified class name as an attribute. Reading then needs neither binding nor spec, but imports a class by a name read from a file, and a renamed or moved class breaks old files.
  4. Read without the type, as plain DataGroup/dict, when the spec is not bound. The value is then not what the workflow returned, so check_fit and typed consumers do not apply.

I lean to 2: it keeps the rule that the spec says what an output is, and separates "can describe" from "can run".

Lingua principale
Python
Stelle
0
Fork
1
Merge medio
8h 48m
PR unite (30g)
70

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di scipp/essapps

Tutte le issue di scipp/essapps

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.