Isolate optional monitoring services from tenant actions on 0.9.0

Read the coordinated changes in apache/openserverless-operator#146 and apache/openserverless-task#239, starting with the operator’s monitoring deployment and the task’s monitoring RBAC. Check the NetworkPolicies, kube-state-metrics permissions, Prometheus exposure, and the upgrade and deletion flows against the stated boundary. Done means both changes ship together and the listed isolation and functional checks pass; an enforcing NetworkPolicy provider is required.

Problem

When optional monitoring is enabled in OPS 0.9.0, ordinary tenant action pods can reach shared Prometheus, kube-state-metrics and Alertmanager listeners. Prometheus additionally uses a fixed NodePort (30299) with its administrative and lifecycle APIs enabled. kube-state-metrics collects Secret metadata and has cluster-wide Secret list/watch privileges.

These are shared administration and observability services. Access by one tenant can affect visibility or availability for other tenants.

Attacker position and risk

The Alertmanager status output in the pinned 0.25 release redacts password and SecretURL fields. Status output alone is not evidence of plaintext Gmail/Slack credential leakage.

Expected behavior and proposed correction

• Remove the Prometheus NodePort and destructive admin API. Preserve its loopback config-reload path and Kubernetes-authorized administrative port-forward access.
• Restrict monitoring ingress with NetworkPolicies. Only same-namespace Prometheus may scrape kube-state-metrics on 8080 or deliver alerts on 9093; Alertmanager gossip is restricted to its peers. Permitted selectors exclude every pod carrying the invoker-controlled user-action-pod label.
• Remove Secret collection and Secret list/watch from kube-state-metrics.
• Apply isolation before workloads, roll legacy pods once, wait for deployment readiness and reconcile on operator resume/configuration update.
• Keep policies until protected pods terminate during explicit component deletion.
• Apply reduced cluster-scoped monitoring RBAC from the administrator-owned task before updating the operator, without granting new operator escalation privileges.

Coordinated PRs targeting 0.9.0: apache/openserverless-operator#146 and apache/openserverless-task#239. Component versions, receiver configuration and application code are preserved. Both changes should ship together.

Validation

Existing synthetic K3s baseline, with the local patched operator:

• 45/45 live boundary and functional checks passed: tenant Service/pod-IP denial, forged trusted labels, foreign namespace denial, trusted scraping and alert delivery, loopback reload, disabled admin API, removed NodePort, reduced Secret RBAC and administrative port-forward.
• Repeated operator resume retained monitored pod identities and readiness.
• Alpha/Beta: 15/15 application checks; Redis Lua: 84/84; Streamer: 24/24. Application source unchanged; scheduled heartbeats continued.
• Existing Kubernetes resource identities and application source hashes were preserved.
• The PR candidate rebased onto the current Apache 0.9.0 base passes 88/88 operator unit tests.
• Focused Daybreak diff reviews completed without reportable findings in the reviewed patch.

Deployment requirements and known limits

An enforcing NetworkPolicy provider is required. Standard Kind networking without policy enforcement does not establish this boundary. Administrator-created broad allow policies and compromised permitted control-plane pods are outside the ordinary tenant model.

Install matching task/operator changes together. An administrator performing an image-only upgrade must first apply the updated monitoring role. This iteration tested an existing-cluster upgrade and real operator resume, not a new full-cluster reinstall or the exact public update command with a released image.

Whole-Whisk owner garbage collection still has no guaranteed policy-last deletion ordering; the existing accepted teardown limitation remains documented. Alertmanager receiver credentials stored in a ConfigMap remain a separate hardening concern for principals already authorized to read cluster configuration.