Roadmap step 5: HTTP transport

Open
#12 15 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
django, python
Domain
api, backend

Research direction

Start with the task 5 files _checks.py, _host.py, and _declaration.py, then read ADR 0014 and the remote-provider sections of specs 02 and 05. Check how task 4 exposes the HTTP transport and how the existing settings and Resolve rows are built. Done means implementing task 5’s listed settings, checks, and resolution behavior, with tests for URL admission, unreachable providers, and the specified refusal cases.

Written by the indexing model from the issue text.

Description

roadmap

Roadmap step 5: the HTTP transport. Provider endpoints served by the host, OpenAPI at the well-known path, the connector over HTTP, shared-secret tokens for the calling app, the principal header, and remote providers under single. Plan per CONTRIBUTING S4 and ADR 0018; it builds on step 4 (#11).

Implements

  • spec 03: the whole document, from outline to requirements: addressing, call context, errors, projection (HTTP-2), calling app and principal, shared-secret tokens, health.
  • spec 02: the network half of the deadline (TRN-4, TRN-38) and of sending at most once (TRN-17); choosing each contract version's one provider under single.
  • spec 05: Resolve's "configured remote provider", and serving while a remote provider is down.
  • spec 06: the contract suite over HTTP, and the conformance items of spec 03.

Depends on

  • Accepted: ADR 0007, 0008, 0010 (narrowed by 0023 Decision 11), 0017, 0019, 0020, 0021, 0022, 0023, 0025. Step 4 (#11), with its PR 9 (#90) merged: task 3 builds on the principal backends.
  • Proposed, each accepted by the first task that builds it: 0011 (#107), 0014 (#106), 0027 (#105), 0028 (#108). Until a record is Accepted, no task builds on it.
  • Avoidable: 0016 (step 6, #13), since only single ships; the repository's open question 7 (production authentication), since only the shared-secret backend ships, for development.

New records (proposed in #105 to #108; where this plan and a record differ, the record wins)

  • Identity on an HTTP call, ADR 0027 (#105; tasks 3, 4, 6): caller backends under NEOLITH["CALLER_BACKENDS"], SharedSecretBackend on the standard library, keys and rotation, the principal header read only after the allow-list, header outcomes, the 401 challenge, refusals before sending. Its Decision 11 narrows 0017 Decision 4 on tracestate.
  • Remote providers, ADR 0014 amended (#106; tasks 4, 5): NEOLITH["REMOTE_PROVIDERS"] keyed by contract identity, the origin url, transport choice at set-up, no network check before serving (an exception to K2, stated in #106), the private over_http test seam.
  • Provider endpoints, ADR 0011 amended (#107; tasks 1, 6, 7): no web framework; the host owns each provided version's prefix; 0017's order on a contract path; one provider side for both transports; responses and their logging; ASGI; the OpenAPI document.
  • HTTP client, ADR 0028 (#108; tasks 2, 4): the standard library's http.client, one deadline for every blocking step, strict responses, a pool per origin, no resend, TLS on the system store.

How the four were reconciled (19 contradictions, each settled by the record that owns the subject) is summarised on this issue's comments.

Public surface

  • NEOLITH["CALLER_BACKENDS"] and NEOLITH["SIGNING_SECRETS"]; neolith.callers.CallerBackend and SharedSecretBackend; neolith.CredentialRefused (0027).
  • NEOLITH["REMOTE_PROVIDERS"]; neolith.transports.HTTPTransport, a name for the setting only (0014).
  • Contract paths and /.well-known/neolith/{openapi.json,health}, with nothing for an app author or deployer to write (0011).
  • Checks, working ids fixed by each PR: neolith.W002 plain http (0014), W003 shadowed URL (0011), W004 no caller backend, W005 a principal backend declaring a neolith- header, W006 (deploy) SharedSecretBackend listed (0027).
  • No runtime dependency. One test-only dependency, trustme (task 2).

Design gate

  • Admitted set: each record's admitted set: the four NEOLITH keys, every header and body form on a contract path, the request the client writes and the responses it accepts. Settings outside it stop the host before Serve with a message naming the input and never a secret or url; requests outside it get spec 03's kind, or no neolith error body off a contract path (APP-1, K2, K9).
  • Source of truth: Django 5.2.19 and 6.1.3 (core/handlers, utils/log.py, views/debug.py, middleware/security.py, test/testcases.py), asgiref 3.13.0, Python 3.12 and 3.13 http/client.py, ssl.py and socket, gunicorn 27.0.1, read as the records cite them.
  • Failure modes: WSGI and ASGI; repeated and underscored headers; a deadline passing in each phase, at a boundary, and with none; 1xx, ambiguous framing and an early refusal with a reset; fork during resolution; secrets and the principal in every output, Django's error mail included; an unreachable provider and one on an older contract version.

Tasks: one PR each

Each PR carries its own spec delta, tests and code (ADR 0018), targets the open release branch, and lists the files it owns. Tasks in one wave touch no common file, apart from the index rows of docs/adr/README.md and one new field each on the set-up table in _host.py's build; whichever lands second merges the release branch. New IDs are named HTTP-new-a and on until the PR numbers them from HTTP-3, TRN-64 and APP-51.

Wave 1, in parallel (after #105 to #108 and #90 merge)

  • 1. One provider side (#122) (refactor/provider-side; _connector.py, _context.py, _crossing.py). From 0017's step 4 on, the in-process transport's provider side becomes shared: it takes a deadline counted at arrival, an optional idempotency key, a principal already read, and a receiver kind that turns on 0007 Decision 3's network receive check (CON-39, CON-40, CON-42). In-process behaviour is unchanged. Covers the receive items left on this issue: exact integer bounds (2**60 + 1 against maximum: 2**60), duplicate member names, lone surrogates, NaN and 1e400. Adds no ID; moves CON-39, CON-40 and CON-42 to this step's network receiver in tests/spec_trace.toml. Verify: the in-process suite unchanged; the network receiver refuses each listed text. Accepts 0011. Gate: provider endpoints.
  • 2. HTTP client (#124) (feat/http-client; new _http.py, pyproject.toml test group). Request writer, strict response reader, deadline-bounded socket wrapper, threaded resolution with partial deadlines, pool per origin with fork reset, TLS context and its test seam; size reported from the spike. Adds TRN-new-a (stop waiting at the deadline in every phase, resolution included), TRN-new-b (no resend; a reused connection that accepted no byte is not a send), HTTP-new-a (Content-Length and Accept-Encoding: identity, never chunked), HTTP-new-b (no redirect, no content coding), HTTP-new-c (read past 1xx other than 101), HTTP-new-d (unavailable on an ambiguous status line, field line, framing or chunk size). Verify: 0028 Decision 9's socket-pair list, including a drip in headers and body cut at the deadline, 103 then 200 on a reused connection, CL with TE, fork with a resolution in flight, and trustme TLS with a CA lacking the AKI. Accepts 0028. Gate: HTTP client.
  • 3. Caller backends and the token (#123) (feat/caller-backends; new callers.py, _token.py; _checks.py, __init__.py, principals.py, one field in _host.py). CallerBackend, SharedSecretBackend (check and mint), CredentialRefused on both sides (request side logged at WARNING, 0023 open question 4), the CALLER_BACKENDS and SIGNING_SECRETS rows, the secret form and key rules, W004 to W006. Adds HTTP-new-e (recognition and abstaining), HTTP-new-f to HTTP-new-m (one per validity item of spec 03's token list), HTTP-new-n (a recognized invalid token fails with no fallback). Verify: spec 03's example token verifies at iat and fails at exp + 60; Hypothesis mutations of one byte, member, type or time give the row they break; hex, padded and short secrets and the example secret in both spellings refused by name; no secret or token in any log line, refusal or error text. Accepts 0027, and adds Narrowed by 0027 (Decision 11) under 0017's status. Gate: identity on an HTTP call.
  • 7. OpenAPI projection (#121) (feat/openapi; new _openapi.py). The 3.1.2 document from Contract.describe() at set-up, as fixed bytes: dialect, no servers, one post per action, operationId, x-neolith-idempotent, keyed components, 422 oneOf, the bearer scheme. Adds HTTP-new for HTTP-2's structure. Verify: the document validates against the OpenAPI 3.1 schema with jsonschema; the round-trip property (CON-16) runs on its components; equal hosts give equal bytes. Accepts nothing; merges after task 1. Gate: provider endpoints.

Wave 2

  • 4. Connector over HTTP (#128) (feat/http-transport; new transports.py, _http_transport.py; _connector.py, _context.py). After tasks 1 to 3. Sent carries the deadline with one budget helper; the transport writes 0027 Decision 9's headers in 0028's order with a fresh token per attempt, reads the budget once a connection is ready, maps responses (200 and 204 paired as 0011 Decision 4 says, the error body's kind, unavailable, internal) and runs the caller's receive check; the private over_http seam. Adds HTTP-new-o (budget read when the request is written), HTTP-new-p (a call with no calling app, or no signing key, refused before sending with the in-process outcome), HTTP-new-q (the kind from the body only; a non-success without a neolith error body is unavailable; a success that is not a valid reply is internal); the spec delta settles CON-28 for an unknown kind whose data is not an object or is absent (left on this issue). Verify: against a local server, each mapping row and the empty-reply pairing; a call from no app gives the in-process message unsent. Accepts 0014. Gate: remote providers, HTTP client.

Wave 3, in parallel

  • 5. Remote providers (feat/remote-providers; _checks.py, _host.py, _declaration.py). After task 4. The REMOTE_PROVIDERS rows, the two Resolve rows, transport choice at set-up, W002, the override_settings refusal, and 0027 Decision 9's set-up refusal for an app with no signing key. Adds TRN-new-c (under single, the one provider: in the host when one exists, else the configured remote one), TRN-new-d (stop before serving on a remote entry no app requires, or one an app provides), APP-new-a (never refuse to serve because a remote provider cannot be reached). Verify: Hypothesis property that a drawn url is admitted exactly when it equals its origin's serialisation; a refused url with a password or a token absent from the message; W002 on foo.localhost and not on loopback; an unreachable provider sets up, boots, serves and gives unavailable. Accepts nothing. Gate: remote providers.
  • 6. Provider endpoints (#129) (feat/provider-endpoints; new _endpoint.py; middleware.py, _boot.py, _server.py, one field in _host.py). After tasks 1, 3 and 7. RootMiddleware answers every owned path; rows 1 to 13 of 0011's table in 0017's order; 0027's steps 1 to 3, header outcomes, tracestate cut and drop, the reporter filter; replies, errors, Allow/Accept-Post/Accept-Encoding; one log line per response marked logged; ASGI per request; the well-known paths with GET and HEAD; builds _openapi.document() at set-up into its _host.py field, where a ValueError stops set-up as a package fault; W003; internal's cause on django.neolith; neolith run refusing a repeated Content-Type or Content-Length. Adds about twenty HTTP IDs, one per rule: authenticate before parsing the body or reading context headers; the unauthenticated cases; Host not served; owned path with no action; method; query; Content-Type; Content-Encoding; Content-Length; body limit (MAY); reply forms; unknown headers and the reserved Neolith- prefix; OWS; tracestate (three); arrival before authentication; the exact 401 challenge; the principal read only after the allow-list; well-known methods; health means ready. Verify: one test per row of 0011's and 0027 Decision 8's tables, under WSGI and ASGI; an unlisted app with a malformed principal gets permission_denied; log_response skips a marked response on Django 5.2 and 6.1; no principal in mail_admins for an internal. Accepts nothing. Gate: provider endpoints, identity on an HTTP call.

Wave 4

  • 8. Over every transport, and the step's documents (feat/http-conformance; tests/ contract suite, tests/spec_trace.toml, docs/django-host.md, docs/roadmap.md, docs/adr/). After tasks 1 to 7. The shared contract suite through over_http against a live server (T2); in-process and HTTP parity for the same call, a call from no app included, on a host that signs and accepts under one key; spec 06's items for spec 03; 03 added to enforce. Documentation: owned paths, W003, decoded paths, the ASGI body bound, response logging, caller backends and WSGIPassAuthorization, REMOTE_PROVIDERS and ALLOWED_HOSTS, calls with no deadline, sync_to_async serialisation, private CAs and their extensions, and the per-call cost of a new connection measured on the reference machine (K10). The roadmap's step 5 row and library paragraph drop django-ninja and httpx; 0011's file is renamed to its title, with every link. Verify: spec_trace.py reports every step 5 ID covered. Accepts nothing. Gate: all four records.

Deferred

  • To step 6 (#13): local-first, failover and retries (TRN-8), more than one provider per identity, a per-try timeout, and the failover test.
  • To later records, each named as an open question in its record: production authentication (open question 7: mTLS, asymmetric tokens, signed delegation, per-provider keys); an async connector and transport; an outbound proxy; a host under a path prefix; a host maximum budget and Drain's bound for inbound calls; a size limit for replies and error bodies (0028 open question 1, which includes this issue's error-body bound); hooks for contract calls; a provisional IANA registration of the well-known name; an unimplemented kind; headers a provider must understand.
Dominant language
Python
Stars
1
Forks
0
Avg merge
5h 30m
Merged PRs (30d)
85

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stack314/django-neolith

All issues in stack314/django-neolith

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.