[Bug] Operator memory usage scales with cluster size instead of with the number of Redkeys (OOMKilled on startup)

Start with cache setup in cmd/main.go and the owned-resource watches in internal/controller/redkey_controller.go; check how internal/controller/redkey_robin.go applies the Robin labels. Add cache scoping and managedFields stripping, then cover labelled and unlabelled objects with unit or envtest tests. Update config/manager/manager.yaml and verify the CSV defaults; done means the acceptance criteria are met, including a safe path for existing unlabeled objects.

Summary

The operator's memory footprint is driven by the total number of Deployments, ServiceAccounts, Roles and RoleBindings in the whole cluster, not by the number of Redkey resources it manages. On a medium/large OpenShift cluster this results in ~430–700 MiB of memory for a single Redkey, and the controller-manager is OOMKilled in a loop with the default resources shipped in the CSV (limits.memory: 300Mi).

Observed behaviour

Environment: OpenShift (Kubernetes v1.33), operator v0.2.1 installed via OLM with an AllNamespaces OperatorGroup (no --watch-namespaces).

• With the default CSV resources (limits: 300m / 300Mi, requests: 10m / 64Mi) the manager pod is OOMKilled shortly after startup and enters CrashLoopBackOff. The CSV loops through Succeeded → Failed (ComponentUnhealthy) → NeedsReinstall → Installing every ~3 minutes.
• After raising the limit to 1Gi (via Subscription.spec.config.resources) the pod becomes stable, with these measurements:

Metric	Value
Redkey resources in the cluster	1
Deployments + ServiceAccounts + Roles + RoleBindings in the cluster	70,590
Memory shortly after startup	690 MiB
Memory at steady state	~431 MiB
CPU at steady state	2m

That's roughly 6 KB per cluster object at steady state and ≥10 KB per object at the startup peak. Almost none of that memory relates to objects the operator owns.

Side effect with OLM: Subscription.spec.config.resources is applied by OLM only after the Subscription's status.installedCSV is populated, so the first Deployment revision always starts with the CSV defaults. On two clusters the operator stayed in an OOM loop for ~5 and ~13 minutes before OLM rolled out the override. Because of this, the CSV defaults must be enough for the operator to start on its own.

Root cause

SetupWithManager (internal/controller/redkey_controller.go) registers owner watches on several core types:

return ctrl.NewControllerManagedBy(mgr).
    For(&redisv1.Redkey{}).
    Owns(&redisv1.RedkeyConfig{}).
    Owns(&appsv1.Deployment{}).
    Owns(&corev1.ServiceAccount{}).
    Owns(&rbacv1.Role{}).
    Owns(&rbacv1.RoleBinding{}).
    ...

and the manager cache (cmd/main.go) is only scoped when --watch-namespaces is set:

var cacheOpts cache.Options
if watchNamespaces != "" {
    ...
    cacheOpts.DefaultNamespaces = defaultNamespaces
}

With cluster-wide scope, each Owns(...) (and every cached client.Get/List on those types) starts a cluster-wide informer with no label selector. The informer lists and keeps in memory every Deployment, ServiceAccount, Role and RoleBinding in the cluster, including metadata.managedFields. OpenShift creates several ServiceAccounts and RoleBindings per namespace (default, builder, deployer, system:image-pullers, …), so on clusters with many namespaces these informers alone hold tens of thousands of objects.

The initial LIST of every informer also causes a large transient allocation spike at startup and on every leader change. That spike sets the minimum memory limit needed for the operator to start.

Notes:

• Memory does not depend on spec.primaries / replicasPerPrimary: StatefulSets, Pods, Services, ConfigMaps, PDBs and PVCs are managed by Robin, not by the operator.
• Per Redkey, the operator caches only ~6 objects (Redkey, 1–2 RedkeyConfigs, robin Deployment/SA/Role/RoleBinding), which is roughly 50–100 KB per Redkey including GC overhead. 800 Redkeys would add only ~40–80 MiB. The cluster-wide baseline is the real problem.

Proposed solution

1. Restrict the cache for owned core types with a label selector (main fix)

Every object the operator creates for Robin already carries the internal labels below, which always override user labels (mergeMeta(..., base) in internal/controller/redkey_robin.go):

ClusterLabel:                   cluster.Name,   // redkey.inditex.dev/cluster
"redkey.inditex.dev/component": "robin",

Use them to scope the informers:

robinSelector := labels.SelectorFromSet(labels.Set{
    "redkey.inditex.dev/component": "robin",
})
// or: an Exists requirement on redkey.inditex.dev/cluster

cacheOpts.ByObject = map[client.Object]cache.ByObject{
    &appsv1.Deployment{}:     {Label: robinSelector},
    &corev1.ServiceAccount{}: {Label: robinSelector},
    &rbacv1.Role{}:           {Label: robinSelector},
    &rbacv1.RoleBinding{}:    {Label: robinSelector},
}

This works together with the existing DefaultNamespaces scoping when --watch-namespaces is set.

Expected result: the cache size depends only on the number of Redkey resources, regardless of cluster size.

2. Strip managedFields from cached objects

cacheOpts.DefaultTransform = cache.TransformStripManagedFields()

managedFields are not used by the controller and often take as much space as the rest of the object.

3. Raise the CSV / manager default resources

Update config/manager/manager.yaml (which ends up in the CSV) so the operator can start without relying on Subscription.spec.config, which OLM applies late. Once (1) and (2) are in place, something like:

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 10m
    memory: 128Mi

Optionally, set GOMEMLIMIT from the container memory limit (resourceFieldRef: limits.memory, with a safety ratio) so the Go GC runs more aggressively before reaching the cgroup limit.

Migration / compatibility considerations

• Objects without the label become invisible to the cache. If any Deployment/SA/Role/RoleBinding created by an older operator version lacks redkey.inditex.dev/component=robin, the cached Get would return NotFound, and the reconciler would try to Create it and fail with AlreadyExists. Options:
  • confirm that every released version has always set these labels, or
  • on AlreadyExists, fall back to mgr.GetAPIReader() to read the object, then patch it with the labels so it shows up in the cache from then on.
• The label must stay immutable and always enforced on owned objects. This is already true because internal labels override spec.labels. Add a comment or test so it isn't accidentally removed.
• Any future Owns(...)/Watches(...) or cached read on new core types should get the same ByObject scoping.

Acceptance criteria

• cache.Options.ByObject scopes Deployment, ServiceAccount, Role and RoleBinding informers using the Robin labels.
• DefaultTransform strips managedFields.
• Unit/envtest test: an unlabelled object of each owned kind is not in the cache, and a labelled one is.
• Migration path for pre-existing objects without labels, or a documented guarantee that they always existed with labels.
• Default manager resources updated in config/manager/manager.yaml / CSV.
• On a cluster with ~70k Deployments/SAs/Roles/RoleBindings, steady-state memory drops from ~430 MiB to tens of MiB, and the operator starts with the new default limits without being OOMKilled.

How to measure

# Cluster baseline that currently drives operator memory
oc get deploy,sa,role,rolebinding -A --no-headers | wc -l

# Operator memory
oc adm top pod -n <operator-namespace>

Prometheus (manager metrics): go_memstats_heap_inuse_bytes, process_resident_memory_bytes, controller_runtime_reconcile_time_seconds.