Backend protocol: name the interface between Client and Backend

Start by reading ess/apps/client.py, ess/apps/backend.py, and the lifetime and accumulator docs named in the issue to map the current interface and its guarantees. The proposal names ess/apps/protocol.py as the destination for BackendProtocol, Entry, Selection, FileSource, and Store; the concrete completion criteria are to type Client against the protocol, move local() beside Backend, and preserve public imports. Resolve the stated wait and watch_files design decisions, then move the rules into protocol docstrings and trim their duplicate documentation.

Make the interface that Client calls on Backend an explicit typing.Protocol, and make it the one home of the rules that clients rely on. It is also the shape of the API that an HTTP server would expose for a hosted backend (#27).

Today

Client calls 27 methods of Backend. All of them take IDs and a client: str, and return records, statuses, or values. Client adds no semantics of its own. It does three things:

• Reshaping: it accepts one, a list, or a dict, and returns the same shape.
• Wrapping: it turns IDs into Record, Stage, and Accumulator objects.
• Shortcuts: compute (submit, then wait), latest, and provenance.

So the protocol exists, but without a name:

• client.py imports the concrete class: from .backend import Backend, Entry, Selection.
• local() lives in client.py, which is why the client module imports the backend at all.
• The rules are spread over:
  • the Client docstrings, which point to docs/developer/lifetimes.md and accumulators.md;
  • the root docs/developer/README.md;
  • ADRs 0002, 0003, 0004, and 0006;
  • the 139-line module docstring of backend.py, which mixes rules with implementation.

Proposal

A BackendProtocol in ess/apps/protocol.py, next to Entry, Selection, and the protocols FileSource and Store. Client.__init__ takes a BackendProtocol. local() moves next to the in-process Backend, and ess.apps keeps re-exporting it.

Group	Methods
Sessions	open_client, close_client
Specs	spec, specs
Requests	submit, cancel, status, wait, failure
History	record, records, changes, plain_request
Values	outputs, describe, readable, persist, release
Stages and accumulators	open_stage, open_accumulator, push, freeze, accumulator_outputs
Files	files, watch_files, file_metadata

Backend.close is the lifecycle of the server, not something a client calls, so it stays off the protocol.

The two sides differ as follows:

	Client	BackendProtocol
Serves	one proposal, one session	many clients and proposals, named by ID
Speaks in	Python objects and shapes	IDs, pydantic models, and values; nothing that could not cross a wire
Promises	ergonomics: shapes and shortcuts	the semantics: when a call returns, what is kept and readable, the order of the change feed, what each call may raise

Rule: anything that changes what happens belongs in the protocol. The client only adapts.

Docs move, they are not copied. The rules from lifetimes.md, accumulators.md, and the client docstrings move into the protocol's docstrings. The Client docstrings then describe shapes and shortcuts and link to the protocol. The backend.py module docstring keeps only how the in-process backend implements the rules. Copying the rules instead would add another place to keep in sync.

What it gives us

• One home for the guarantees. App work needs only the client and the protocol, not the backend docs.
• Contract changes are their own diff. changes and describe (#156) touched the client, the backend, views.py, ADR 0004, the README, and the app together. With a protocol, the new methods and their promises are a small diff to review, apart from the implementation behind them.
• The seams of the backend are all named. Up to the client is the protocol. Down are FileSource and Store. Sideways is Binding from essspec.

What it does not give: it does not make backend.py smaller. That is a separate split. Test fakes of the backend would add little, since local() is cheap.

Toward an HTTP wrapper for a hosted backend

The protocol is the server's API surface. A hosted backend is then built as follows:

• Server: a thin layer that maps each route to a protocol call on an in-process Backend.
• Client side: a RemoteBackend that implements the protocol by making HTTP calls. connect(url, proposal) is Client(RemoteBackend(url), ...), and Client does not change.
• Tests: the client and story tests run against both local() and Client(RemoteBackend(test_server(Backend()))), through one parametrized fixture. That shows the two deployments behave the same, which docs/developer/README.md promises ("the caller sees the same records and outputs either way").

Writing the protocol now makes every method's types say what crosses the wire. These methods would need a decision:

Method	In-process today	Over HTTP
open_client(proposal, submitter)	the caller names the submitter	the submitter comes from authentication, and access to the proposal is checked. The returned ID becomes a session with a lease (#27)
every method's client: str	an argument	from the session or token
spec, specs	return WorkflowSpec with live pydantic models	return SpecId and SerializedWorkflowSpec (#88). A Python client maps the SpecId to installed spec modules (ADR 0007)
submit(entries), push(rows), open_stage, open_accumulator	Request.params and rows may hold pydantic model instances	encoded through the params schema and request types (#88, #140)
wait(ids)	blocks without a timeout	blocking does not suit HTTP: make it a long poll with a timeout, with the client looping (#81: timeout= is undecided)
watch_files(selector)	returns an iterator	an iterator does not cross a wire: give it a cursor form like changes, and let the client build the iterator (new-file signal in #155)
outputs, accumulator_outputs	return the value the workflow returned, not a copy	a copy, through a codec per output type (#152); describe and select= (#149) limit what is sent
SubmitError (with FieldErrors), NotReadable, RecordFailed, ClientEnded	Python exceptions	part of the contract: a structured error body each
bindings, local(bind=...)	code from the notebook	not part of the protocol: a hosted backend finds bindings through entry points (ADR 0007)

This suggests one more rule: protocol methods are request and response; whatever blocks or iterates is built in Client on top. Today wait and watch_files break this rule.

Not settled by the protocol: authentication and access across proposals, leases and memory caps (#27), wire format and codecs (#152), and versioning of the HTTP API.

Steps

1. Add protocol.py with BackendProtocol, and move Entry, Selection, FileSource, and Store there. Type Client against the protocol. Move local() next to Backend. Public imports do not change.
2. Move the rules into the protocol's docstrings, and trim the client and backend docstrings and the docs pages that held them.
3. Decide wait and watch_files (request/response, with the client looping).
4. Later, with #27: RemoteBackend, the HTTP server, and the parametrized test fixture.

This goes with splitting ess.apps into subpackages (client/, backend/, files/, ui/, panel/, automatic/), with the layering checked by import-linter. That split is a separate issue.