kubernetes/kubernetes

Prevent duplicate initializations of cache and etcd store

Open

#133877 opened on Sep 3, 2025

View on GitHub
 (4 comments) (0 reactions) (1 assignee)Go (122,268 stars) (43,066 forks)batch import
help wantedkind/bugsig/api-machinerytriage/accepted

Description

What happened?

When working on disabling stats collection for events I noticed higher than expected number of calls to getKeys. I traced it down to apiserver starting two separate storage instances for events.

Similar to previous watch cache issue with HPA https://github.com/kubernetes/kubernetes/pull/132924 this is wasting resources and causes problems when storage code assumes running as singleton. With introduction of resource object estimation in etcd store we double amount of memory used for the mechanism.

We have duplicated initialization for serviceaccounts and events. We can address the specific cases, still the possibility of duplicates remain.

One solution would be to introduce a integration/e2e tests that validates number of storage initializations. This sounds like a simple mitigation that would not force us to rewrite the whole storage initialization.

/cc @liggitt @wojtek-t

What did you expect to happen?

There should be only one instance of cacher and storage running for resource.

How can we reproduce it (as minimally and precisely as possible)?

To find cases I added logs to constructor of etcd and cacher and grepped logs:

	klog.InfoS("cacher.NewCacherFromConfig", "group", config.GroupResource.Group, "resource", config.GroupResource.Resource, "resourcePrefix", config.ResourcePrefix)
$ kubectl logs -n kube-system kube-apiserver-kind-control-plane | grep "New" | cut -d' ' -f12- | sort | uniq -c | grep 2
      2 cacher.NewCacherFromConfig" group="" resource="serviceaccounts" resourcePrefix="/serviceaccounts"
      2 etcd3.New" group="" resource="events" resourcePrefix="/events"
      2 etcd3.New" group="" resource="serviceaccounts" resourcePrefix="/serviceaccounts"

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

Contributor guide