Streamer 0.9.0 hardening: authenticated callbacks and backward-incompatible client migration

Start with the listed Go handlers and TCP callback server at the assessed Streamer commit, then review the migration proposal and remaining validation gates. The work spans callback authentication, input limits, lifecycle bounds, quotas, and a breaking producer and consumer protocol, so coordinate a release decision and migration plan with maintainers. Done means an independently reviewed implementation, migration tests for affected producers and consumers, and published protocol and rollout guidance.

Summary

The Streamer callback contract documented in #82 requires a coordinated client migration before introducing callback authentication and bounded resource handling. This issue proposes that migration and the associated regression coverage for the 0.9.0 Streamer.

This is an engineering and compatibility discussion. Operational security reproductions, detailed attack sequences and private lab evidence are intentionally omitted; undisclosed vulnerability details should follow the ASF security reporting process.

Assessed source and scope

The inspected Apache Streamer 0.9.0 branch is at 2de92993ff4930715eec38ae80ed91b9ded15c21. Relevant source:

• HTTP web-action handler
• Private-action handler
• Request-body helpers
• TCP callback server

Four areas were identified in that baseline:

1. Asynchronous lifecycle and ownership: cancellation, channel ownership and buffer ownership need to prevent process-level failure, blocked work and corrupted stream output.
2. Callback identity: a callback needs authentication bound to its individual invocation. Authorizing the HTTP action invocation does not establish the identity of a separate TCP producer.
3. Input handling: JSON size and object shape need explicit validation before callback allocation and action invocation.
4. Resource lifetime: upstream requests, callback establishment, idle connections, downstream writes and cancellation need bounded lifetimes and reliable cleanup.

Two additional availability findings were identified in the first hardening candidate, not in the unmodified Apache 0.9.0 implementation: an undifferentiated global admission limit and serialized authentication of partial callback connections. They are listed separately because they must not be described as two additional vulnerabilities originally present in Apache 0.9.0.

Attacker and risk matrix

Actor	Required access	Risk within the assessed scope	Important limits
Unauthenticated HTTP caller	Access to an enabled public web-action stream whose application permits the invocation	Streamer availability through input/lifecycle handling	An ingress being reachable does not make every application action anonymous; its authorization policy still applies.
Ordinary tenant developer	Credentials for their own namespace and permission to invoke/deploy their own actions	Availability of the shared Streamer; the first candidate also needed admission fairness between tenants	No Kubernetes or sudo access is assumed. This does not demonstrate authorization bypass into another namespace.
Code running in a tenant action	Network reachability to another callback and knowledge of its address/port	Stream integrity and availability if the callback lacks invocation-bound authentication	Internal reachability and callback discovery are prerequisites, not assumed internet capabilities. Privileged lab instrumentation is not proof of tenant discovery.
Internet-only caller without internal network reachability	HTTP ingress only	Public HTTP handling remains relevant	Direct access to another tenant's TCP callback has not been established for this actor.
Kubernetes administrator or host sudo user	Administrative control	Outside the tenant-isolation threat model	Existing administrative powers are not evidence of a Streamer privilege escalation.

This assessment does not establish arbitrary code execution, database takeover or data exfiltration through the Streamer. Risk depends on deployed routes, application authorization, network isolation and external rate limits; no CVE, CVSS score or fixed Apache release is claimed here.

Proposed implementation

• Use request-scoped cancellation, explicit channel ownership, owned payload buffers, and bounded upstream response handling.
• Bound request JSON before invoking an action; reject oversized or invalid input consistently.
• Give each invocation a fresh callback capability and authenticate it before forwarding any callback payload. Do not log or return the capability to the browser.
• Authenticate pending callback peers concurrently within a fixed resource bound; accept only one valid producer and close pending peers on cancellation.
• Keep a global stream bound and add verified-caller/target quotas so one tenant cannot consume all capacity. Do not derive identity from arbitrary Authorization text or spoofable forwarded headers. Preserve custom web-action authorization.
• Document body limits, start/idle/total deadlines, quotas and returned errors as part of the user-facing contract.

The current local candidate uses 1 MiB JSON, 64 streams per process, eight per verified caller/target, bounded anonymous capacity, a 20-second callback-start deadline, one-second preamble deadline, 30-second callback-idle deadline and five-minute total lifetime. These are proposed implementation choices, not existing Apache defaults. Anonymous clients behind the same ingress may share a direct-peer quota. Per-process bounds do not replace deployment-level rate limiting.

Breaking changes: existing streaming actions must be updated

Replacing only the shared Streamer image would break existing raw-TCP streaming actions.

The existing contract injects STREAM_HOST and STREAM_PORT. The authenticated candidate additionally injects a per-invocation STREAM_TOKEN and requires the producer to send:

OPS-STREAM <STREAM_TOKEN>\n

The proxy consumes this preamble and forwards only the following payload bytes. A legacy producer sending its old payload immediately will not authenticate. The patched server must not silently fall back to accepting unauthenticated callbacks.

There is also a framing change: the old proxy adds newlines at TCP-read boundaries. The candidate relays payload bytes unchanged. A TCP read or an HTTP chunk is not an application record; producers using NDJSON must supply their own record delimiters, and browser clients must buffer incomplete records and UTF-8 sequences.

Migration proposal

1. Inventory every producer and consumer using the shared Streamer, including examples and custom actions.
2. Update producers first: when STREAM_TOKEN is supplied, send the authenticated preamble immediately after connecting, before waiting for model output; otherwise use the legacy callback contract. Keep the token out of logs, persistent configuration and browser responses. The old proxy's framing limitations still require testing; this transition is not a blanket compatibility guarantee.
3. Update consumers to parse records independently of network chunk boundaries, and test errors, Stop/cancellation, model startup and long-lived streams.
4. Update the shared proxy only after every affected producer is ready. If coordinated migration is impossible, use an explicitly separated rollout with routing reviewed by the maintainers; do not enable an insecure fallback in the authenticated instance.
5. Publish protocol documentation, examples, release notes, limit changes and rollback instructions together. Downgrading the proxy also restores the behavior the hardening is intended to address.

Maintainer decision requested: should this intentionally incompatible protocol be introduced in a subsequent feature release, or through an explicitly coordinated 0.9.0 maintenance change? It should not be presented as a transparent image-only upgrade.

Local validation and remaining gates

The local candidate has passed Go tests with the race detector and go vet, bounded callback/concurrency/cleanup regressions, and Alpha/Beta HTTP/TCP and application checks on K3s. TruChat protocol/parser and real-proxy smoke tests passed with synthetic AI/controller responses; this is not a full deployed-model acceptance test.

Remaining gates include a focused independent security review of the final combined patch, migration testing for other deployed producers/consumers, and the maintainers' release/compatibility decision. No upstream Streamer PR, Apache fixed release or complete absence of vulnerabilities is asserted.

Follow-up candidate finding references

These identifiers refer only to the reviewed first-candidate snapshot, not to an immutable Apache source commit:

• Canonical finding ID: csf_acd7a67907dd6b97edc571db; primary fingerprint: codex-security/v1:sha256:32b6556c6c562421235b27caf4bb91efcfe5a9d5525c32c11f804dd419fc5b14. Candidate locations: entrypoint src/httpserver.go:73-81; root control src/handlers/stream.go:39; sink src/handlers/stream.go:71-77; outcome src/handlers/stream.go:124-144.
• Canonical finding ID: csf_44edad4853afa81e5093319a; primary fingerprint: codex-security/v1:sha256:9e039686fd2cd22697227a741acd620dd1d317fbbf05c49cbbe2349d7a947d4c. Candidate locations: root control src/tcp/tcpserver.go:74-89; sink src/handlers/stream.go:124-144; evidence src/tcp/security_test.go:70-92.