Established connections are dropped when cluster-autoscaler scales down a load balancer backend node

Aperta
  • Qualcosa non va?
#554 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
28/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
cloud, networking

Direzione di ricerca

Start with getBackends in pkg/cloudprovider/providers/oci/load_balancer_spec.go, where backends are rebuilt from the remaining nodes, and the NLB client code that drops IsDrain. The proposed change keeps a backend whose Node still has the ToBeDeletedByClusterAutoscaler taint but sets isDrain: true, and removes it once the Node is gone. Done means the opt-in annotation works and a test shows the drained backend is restored when the taint is removed. Confirm the approach with maintainers before coding.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT (with a proposed fix)

Versions

CCM Version: v1.32.2 (also seen on v1.31.2; behavior is the same on master at 93969b5)

Environment:

  • Kubernetes version: v1.32.13
  • Others: self-managed clusters on OCI (Cluster API), cluster-autoscaler v1.31.5 (clusterapi provider), OCI Network Load Balancers with externalTrafficPolicy: Local

What happened?

When cluster-autoscaler scales down a node, it first adds the ToBeDeletedByClusterAutoscaler taint and then evicts pods. The generic service controller in k8s.io/cloud-provider excludes tainted nodes from every LoadBalancer Service (stableNodeSetPredicates / nodeUnTaintedPredicate). On the next sync of the Service, the OCI CCM rebuilds the backend set from the remaining nodes (getBackends) and the update deletes the tainted node's backend. OCI removes the backend immediately, and established connections through that node stop receiving packets with no reset. Clients hang until their own timeouts.

This happens while the node and its pods are still running. In our clusters the CCM typically removes the backend within seconds of the taint, because other node churn triggers a sync, while cluster-autoscaler is still evicting pods for another minute or two. Clients holding keep-alive connections through that node (for example HTTP connection pools to an ingress controller) see requests hang until their timeouts, and the stuck connections back up their pools. On busy clusters this happens on most scale-downs of a node carrying load balancer traffic.

The k8s code acknowledges the gap: the comment on stableNodeSetPredicates says tainted nodes are excluded on the next sync "and cause connections on said node to not connection drain". It was discussed upstream as a provider concern (KEP-3458, KEP-3836, and kubernetes/kubernetes#129605, where SIG Cloud Provider had no generic fix).

For comparison, the AWS cloud controller manager leaves draining to the load balancer: for NLBs it deregisters targets (ensureTargetGroupTargets), AWS keeps existing flows during the target group's deregistration delay (300s by default), and the CCM treats targets in DeregistrationInProgress as already gone (L930). For Classic ELBs it exposes service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled and -connection-draining-timeout (L130-L134). OCI backends support the same idea through isDrain ("stops receiving new connections but continues to receive traffic on existing connections"), but the CCM never sets it: removed backends are always deleted outright.

We also confirmed that an unhealthy NLB backend keeps forwarding established connections; only removing the backend cuts them. So the health check failing early (KEP-3836) is not the problem; the deletion is.

What you expected to happen?

A node that is being scaled down keeps its existing load balancer connections until the node is actually gone, the same as on AWS: the backend is drained (no new connections) instead of deleted, and removed once the node no longer exists.

How to reproduce it (as minimally and precisely as possible)?

  1. Create a type: LoadBalancer Service (NLB, externalTrafficPolicy: Local or Cluster) and hold a long-lived TCP connection through one backend node (e.g. HTTP keep-alive sending a request every second).
  2. Add the cluster-autoscaler taint to that node by hand: kubectl taint node <node> ToBeDeletedByClusterAutoscaler=<unix-ts>:NoSchedule
  3. Trigger a sync of the Service (any change to its annotations, or any node add/remove in the cluster).
  4. The CCM removes the node's backend and the long-lived connection hangs, even though the node and the pod behind it are still healthy.

Anything else we need to know?

Proposed fix, opt-in per Service so default behavior doesn't change:

  • New annotations oci-network-load-balancer.oraclecloud.com/drain-backends-on-scale-down: "true" (NLB) and service.beta.kubernetes.io/oci-load-balancer-drain-backends-on-scale-down: "true" (LB).
  • When a backend is missing from the desired set but its Node still exists with the ToBeDeletedByClusterAutoscaler taint, keep the existing backend unchanged except isDrain: true. If the autoscaler aborts the scale-down and removes the taint, the backend is un-drained. Any other missing backend (Node deleted, exclude label, node selector change) is removed as today.
  • No timers or extra state: the drain decision is recomputed from the Node object on every sync, using the CCM's existing node lister. The NLB client also needs to read and send IsDrain, which it currently drops.

If you agree this should be fixed and are able to prioritize reviewing it, I'm willing to open a PR with this approach and sign the OCA.

Lingua principale
Go
Stelle
160
Fork
109
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di oracle/oci-cloud-controller-manager

Tutte le issue di oracle/oci-cloud-controller-manager

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.