kubernetes-sigs/cluster-api

High memory usage on ClusterResourceSet controllers

Open

#9,843 opened on Dec 11, 2023

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Go (1,532 forks)auto 404
area/clusterresourcesetarea/machinedeploymenthelp wantedkind/bugpriority/important-longtermtriage/accepted

Repository metrics

Stars
 (4,267 stars)
PR merge metrics
 (PR metrics pending)

Description

What steps did you take and what happened?

On a scale test we've been running, we have realized that ClusterResourceSet is consuming a large amount of memory from the environment.

The environment:

  • 200 clusters created
  • Every cluster contains 3 ClusterResourceSet (CNI = Antrea, CSI = vsphere-csi and CPI=vsphere-cloud-provider), with spec.strategy=Reconcile

The growth of memory is exponential with the cluster creation. Following is attached an extraction of the pprof flamegraph from the time the problem happened

What did you expect to happen?

  • First of all, to check if this is the expected behavior
  • If not, that the memory growth is not exponential with the ClusterResourceSet usage

Cluster API version

CAPI 1.5.4, CAPV 1.8.4

Kubernetes version

No response

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

Contributor guide

Research direction
Analyze the pprof flamegraph to identify memory allocation hotspots in the ClusterResourceSet controller. Trace the reconciliation logic to find where memory grows exponentially with cluster count.
Tech stack
go
Domain
backendinfrastructure
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
GoKubernetescluster api
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20