Vector sidecar does not terminate when the main container is OOMKilled (run-to-completion pods)

Description

The shared Vector logging sidecar is shut down by a marker file written from an entrypoint post-hook (create_vector_shutdown_file_command / vector_container in product_logging). The post-hook only runs after the main process exits normally. When the main container is OOMKilled, the kernel delivers SIGKILL to the whole container cgroup (the entrypoint script included), so the post-hook never runs, the marker is never written, and the Vector sidecar keeps running forever.

For run-to-completion pods this prevents the Pod from ever reaching a terminal phase (Succeeded/Failed), because a Pod with restartPolicy: Never only completes once all its containers have terminated. Downstream consumers that poll the Pod phase never observe the failure.

The Vector container and its shutdown mechanism are provided by the shared product_logging framework (in operator-rs) and reused by all operators.

Impact

Affects operators that run Vector on pods which are expected to terminate:

• spark-k8s-operator — driver and executor pods. The driver case also leaves SparkApplication.status.phase stuck on Running.
• airflow-operator — KubernetesExecutor task pods. Airflow's own executor watches the task Pod phase to detect completion, so a stuck pod means the task never resolves.

Not affected: long-running services (StatefulSets/Deployments) — an OOM there restarts the container (restartPolicy: Always) and a running Vector is expected. Short-lived jobs without a Vector sidecar are also unaffected.

Root cause

• product_logging::framework::vector_container runs Vector as a regular container that only exits when the shutdown marker file appears.
• The marker is written by the consumer's entrypoint post-hook, which cannot run on SIGKILL. SIGKILL is uncatchable, so no entrypoint-side change can fix this.

Proposed solution

Model the Vector container as a Kubernetes native sidecar (an init container with restartPolicy: Always). The kubelet then terminates native sidecars automatically once the main containers exit — including on OOM — via SIGTERM, respecting terminationGracePeriodSeconds. Native sidecars also do not hold the Pod in Running, so run-to-completion pods reach a terminal phase on their own and the marker/post-hook mechanism becomes unnecessary.

Availability: native sidecars are beta and enabled by default since Kubernetes 1.29 and GA since 1.33 — covers the supported matrix (Kubernetes 1.31+, OpenShift 4.18+).

Considerations

• Shared blast radius. vector_container is used by all operators; the change and its rollout/testing must cover every consumer, not just the two affected ones.
• Log flushing. Today the post-hook does a short sleep before writing the marker so Vector can flush. With a native sidecar, Vector receives SIGTERM and must flush within the grace period; the marker/_STACKABLE_POST_HOOK path for Vector then becomes redundant and should be removed.
• Backwards compatibility. Decide whether this is gated/opt-in during transition or a straight switch, given the shared surface.

Alternative (for Spark NOT for Airflow)

Each affected operator could instead derive the terminal state from the main container's status rather than the whole-Pod phase. This works for spark-k8s (which reconciles the driver Pod) but not for Airflow, whose task pods are managed by Airflow rather than the operator. It is at best a per-operator stopgap and does not fix the shared root cause.