Backend protocol: name the interface between Client and Backend

Abierto
  • ¿Algo no está bien?
#157 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
40/100
Tipo de issue
Refactorización
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, backend-api-design

Línea de trabajo

Start by reading ess/apps/client.py, ess/apps/backend.py, and the lifetime and accumulator docs named in the issue to map the current interface and its guarantees. The proposal names ess/apps/protocol.py as the destination for BackendProtocol, Entry, Selection, FileSource, and Store; the concrete completion criteria are to type Client against the protocol, move local() beside Backend, and preserve public imports. Resolve the stated wait and watch_files design decisions, then move the rules into protocol docstrings and trim their duplicate documentation.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Make the interface that Client calls on Backend an explicit typing.Protocol, and make it the one home of the rules that clients rely on. It is also the shape of the API that an HTTP server would expose for a hosted backend (#27).

Today

Client calls 27 methods of Backend. All of them take IDs and a client: str, and return records, statuses, or values. Client adds no semantics of its own. It does three things:

  • Reshaping: it accepts one, a list, or a dict, and returns the same shape.
  • Wrapping: it turns IDs into Record, Stage, and Accumulator objects.
  • Shortcuts: compute (submit, then wait), latest, and provenance.

So the protocol exists, but without a name:

  • client.py imports the concrete class: from .backend import Backend, Entry, Selection.
  • local() lives in client.py, which is why the client module imports the backend at all.
  • The rules are spread over:
    • the Client docstrings, which point to docs/developer/lifetimes.md and accumulators.md;
    • the root docs/developer/README.md;
    • ADRs 0002, 0003, 0004, and 0006;
    • the 139-line module docstring of backend.py, which mixes rules with implementation.

Proposal

A BackendProtocol in ess/apps/protocol.py, next to Entry, Selection, and the protocols FileSource and Store. Client.__init__ takes a BackendProtocol. local() moves next to the in-process Backend, and ess.apps keeps re-exporting it.

Group Methods
Sessions open_client, close_client
Specs spec, specs
Requests submit, cancel, status, wait, failure
History record, records, changes, plain_request
Values outputs, describe, readable, persist, release
Stages and accumulators open_stage, open_accumulator, push, freeze, accumulator_outputs
Files files, watch_files, file_metadata

Backend.close is the lifecycle of the server, not something a client calls, so it stays off the protocol.

The two sides differ as follows:

Client BackendProtocol
Serves one proposal, one session many clients and proposals, named by ID
Speaks in Python objects and shapes IDs, pydantic models, and values; nothing that could not cross a wire
Promises ergonomics: shapes and shortcuts the semantics: when a call returns, what is kept and readable, the order of the change feed, what each call may raise

Rule: anything that changes what happens belongs in the protocol. The client only adapts.

Docs move, they are not copied. The rules from lifetimes.md, accumulators.md, and the client docstrings move into the protocol's docstrings. The Client docstrings then describe shapes and shortcuts and link to the protocol. The backend.py module docstring keeps only how the in-process backend implements the rules. Copying the rules instead would add another place to keep in sync.

What it gives us

  • One home for the guarantees. App work needs only the client and the protocol, not the backend docs.
  • Contract changes are their own diff. changes and describe (#156) touched the client, the backend, views.py, ADR 0004, the README, and the app together. With a protocol, the new methods and their promises are a small diff to review, apart from the implementation behind them.
  • The seams of the backend are all named. Up to the client is the protocol. Down are FileSource and Store. Sideways is Binding from essspec.

What it does not give: it does not make backend.py smaller. That is a separate split. Test fakes of the backend would add little, since local() is cheap.

Toward an HTTP wrapper for a hosted backend

The protocol is the server's API surface. A hosted backend is then built as follows:

  • Server: a thin layer that maps each route to a protocol call on an in-process Backend.
  • Client side: a RemoteBackend that implements the protocol by making HTTP calls. connect(url, proposal) is Client(RemoteBackend(url), ...), and Client does not change.
  • Tests: the client and story tests run against both local() and Client(RemoteBackend(test_server(Backend()))), through one parametrized fixture. That shows the two deployments behave the same, which docs/developer/README.md promises ("the caller sees the same records and outputs either way").

Writing the protocol now makes every method's types say what crosses the wire. These methods would need a decision:

Method In-process today Over HTTP
open_client(proposal, submitter) the caller names the submitter the submitter comes from authentication, and access to the proposal is checked. The returned ID becomes a session with a lease (#27)
every method's client: str an argument from the session or token
spec, specs return WorkflowSpec with live pydantic models return SpecId and SerializedWorkflowSpec (#88). A Python client maps the SpecId to installed spec modules (ADR 0007)
submit(entries), push(rows), open_stage, open_accumulator Request.params and rows may hold pydantic model instances encoded through the params schema and request types (#88, #140)
wait(ids) blocks without a timeout blocking does not suit HTTP: make it a long poll with a timeout, with the client looping (#81: timeout= is undecided)
watch_files(selector) returns an iterator an iterator does not cross a wire: give it a cursor form like changes, and let the client build the iterator (new-file signal in #155)
outputs, accumulator_outputs return the value the workflow returned, not a copy a copy, through a codec per output type (#152); describe and select= (#149) limit what is sent
SubmitError (with FieldErrors), NotReadable, RecordFailed, ClientEnded Python exceptions part of the contract: a structured error body each
bindings, local(bind=...) code from the notebook not part of the protocol: a hosted backend finds bindings through entry points (ADR 0007)

This suggests one more rule: protocol methods are request and response; whatever blocks or iterates is built in Client on top. Today wait and watch_files break this rule.

Not settled by the protocol: authentication and access across proposals, leases and memory caps (#27), wire format and codecs (#152), and versioning of the HTTP API.

Steps

  1. Add protocol.py with BackendProtocol, and move Entry, Selection, FileSource, and Store there. Type Client against the protocol. Move local() next to Backend. Public imports do not change.
  2. Move the rules into the protocol's docstrings, and trim the client and backend docstrings and the docs pages that held them.
  3. Decide wait and watch_files (request/response, with the client looping).
  4. Later, with #27: RemoteBackend, the HTTP server, and the parametrized test fixture.

This goes with splitting ess.apps into subpackages (client/, backend/, files/, ui/, panel/, automatic/), with the layering checked by import-linter. That split is a separate issue.

Lenguaje dominante
Python
Estrellas
0
Forks
1
Merge medio
15 h 40 min
PR fusionados (30 d)
59

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de scipp/essapps

Todos los issues de scipp/essapps

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.