Backend protocol: name the interface between Client and Backend

Aperta
  • Qualcosa non va?
#157 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
40/100
Tipo di issue
Refactoring
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, backend-api-design

Direzione di ricerca

Start by reading ess/apps/client.py, ess/apps/backend.py, and the lifetime and accumulator docs named in the issue to map the current interface and its guarantees. The proposal names ess/apps/protocol.py as the destination for BackendProtocol, Entry, Selection, FileSource, and Store; the concrete completion criteria are to type Client against the protocol, move local() beside Backend, and preserve public imports. Resolve the stated wait and watch_files design decisions, then move the rules into protocol docstrings and trim their duplicate documentation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Make the interface that Client calls on Backend an explicit typing.Protocol, and make it the one home of the rules that clients rely on. It is also the shape of the API that an HTTP server would expose for a hosted backend (#27).

Today

Client calls 27 methods of Backend. All of them take IDs and a client: str, and return records, statuses, or values. Client adds no semantics of its own. It does three things:

  • Reshaping: it accepts one, a list, or a dict, and returns the same shape.
  • Wrapping: it turns IDs into Record, Stage, and Accumulator objects.
  • Shortcuts: compute (submit, then wait), latest, and provenance.

So the protocol exists, but without a name:

  • client.py imports the concrete class: from .backend import Backend, Entry, Selection.
  • local() lives in client.py, which is why the client module imports the backend at all.
  • The rules are spread over:
    • the Client docstrings, which point to docs/developer/lifetimes.md and accumulators.md;
    • the root docs/developer/README.md;
    • ADRs 0002, 0003, 0004, and 0006;
    • the 139-line module docstring of backend.py, which mixes rules with implementation.

Proposal

A BackendProtocol in ess/apps/protocol.py, next to Entry, Selection, and the protocols FileSource and Store. Client.__init__ takes a BackendProtocol. local() moves next to the in-process Backend, and ess.apps keeps re-exporting it.

Group Methods
Sessions open_client, close_client
Specs spec, specs
Requests submit, cancel, status, wait, failure
History record, records, changes, plain_request
Values outputs, describe, readable, persist, release
Stages and accumulators open_stage, open_accumulator, push, freeze, accumulator_outputs
Files files, watch_files, file_metadata

Backend.close is the lifecycle of the server, not something a client calls, so it stays off the protocol.

The two sides differ as follows:

Client BackendProtocol
Serves one proposal, one session many clients and proposals, named by ID
Speaks in Python objects and shapes IDs, pydantic models, and values; nothing that could not cross a wire
Promises ergonomics: shapes and shortcuts the semantics: when a call returns, what is kept and readable, the order of the change feed, what each call may raise

Rule: anything that changes what happens belongs in the protocol. The client only adapts.

Docs move, they are not copied. The rules from lifetimes.md, accumulators.md, and the client docstrings move into the protocol's docstrings. The Client docstrings then describe shapes and shortcuts and link to the protocol. The backend.py module docstring keeps only how the in-process backend implements the rules. Copying the rules instead would add another place to keep in sync.

What it gives us

  • One home for the guarantees. App work needs only the client and the protocol, not the backend docs.
  • Contract changes are their own diff. changes and describe (#156) touched the client, the backend, views.py, ADR 0004, the README, and the app together. With a protocol, the new methods and their promises are a small diff to review, apart from the implementation behind them.
  • The seams of the backend are all named. Up to the client is the protocol. Down are FileSource and Store. Sideways is Binding from essspec.

What it does not give: it does not make backend.py smaller. That is a separate split. Test fakes of the backend would add little, since local() is cheap.

Toward an HTTP wrapper for a hosted backend

The protocol is the server's API surface. A hosted backend is then built as follows:

  • Server: a thin layer that maps each route to a protocol call on an in-process Backend.
  • Client side: a RemoteBackend that implements the protocol by making HTTP calls. connect(url, proposal) is Client(RemoteBackend(url), ...), and Client does not change.
  • Tests: the client and story tests run against both local() and Client(RemoteBackend(test_server(Backend()))), through one parametrized fixture. That shows the two deployments behave the same, which docs/developer/README.md promises ("the caller sees the same records and outputs either way").

Writing the protocol now makes every method's types say what crosses the wire. These methods would need a decision:

Method In-process today Over HTTP
open_client(proposal, submitter) the caller names the submitter the submitter comes from authentication, and access to the proposal is checked. The returned ID becomes a session with a lease (#27)
every method's client: str an argument from the session or token
spec, specs return WorkflowSpec with live pydantic models return SpecId and SerializedWorkflowSpec (#88). A Python client maps the SpecId to installed spec modules (ADR 0007)
submit(entries), push(rows), open_stage, open_accumulator Request.params and rows may hold pydantic model instances encoded through the params schema and request types (#88, #140)
wait(ids) blocks without a timeout blocking does not suit HTTP: make it a long poll with a timeout, with the client looping (#81: timeout= is undecided)
watch_files(selector) returns an iterator an iterator does not cross a wire: give it a cursor form like changes, and let the client build the iterator (new-file signal in #155)
outputs, accumulator_outputs return the value the workflow returned, not a copy a copy, through a codec per output type (#152); describe and select= (#149) limit what is sent
SubmitError (with FieldErrors), NotReadable, RecordFailed, ClientEnded Python exceptions part of the contract: a structured error body each
bindings, local(bind=...) code from the notebook not part of the protocol: a hosted backend finds bindings through entry points (ADR 0007)

This suggests one more rule: protocol methods are request and response; whatever blocks or iterates is built in Client on top. Today wait and watch_files break this rule.

Not settled by the protocol: authentication and access across proposals, leases and memory caps (#27), wire format and codecs (#152), and versioning of the HTTP API.

Steps

  1. Add protocol.py with BackendProtocol, and move Entry, Selection, FileSource, and Store there. Type Client against the protocol. Move local() next to Backend. Public imports do not change.
  2. Move the rules into the protocol's docstrings, and trim the client and backend docstrings and the docs pages that held them.
  3. Decide wait and watch_files (request/response, with the client looping).
  4. Later, with #27: RemoteBackend, the HTTP server, and the parametrized test fixture.

This goes with splitting ess.apps into subpackages (client/, backend/, files/, ui/, panel/, automatic/), with the layering checked by import-linter. That split is a separate issue.

Lingua principale
Python
Stelle
0
Fork
1
Merge medio
15h 40m
PR unite (30g)
59

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di scipp/essapps

Tutte le issue di scipp/essapps

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.