High memory usage on ClusterResourceSet controllers

What steps did you take and what happened?

On a scale test we've been running, we have realized that ClusterResourceSet is consuming a large amount of memory from the environment.

The environment:

• 200 clusters created
• Every cluster contains 3 ClusterResourceSet (CNI = Antrea, CSI = vsphere-csi and CPI=vsphere-cloud-provider), with spec.strategy=Reconcile

The growth of memory is exponential with the cluster creation. Following is attached an extraction of the pprof flamegraph from the time the problem happened

What did you expect to happen?

• First of all, to check if this is the expected behavior
• If not, that the memory growth is not exponential with the ClusterResourceSet usage

Cluster API version

CAPI 1.5.4, CAPV 1.8.4

Kubernetes version

No response

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.