Established connections are dropped when cluster-autoscaler scales down a load balancer backend node

Open
  • Is something wrong?
#554 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
28/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
cloud, networking

Research direction

Start with getBackends in pkg/cloudprovider/providers/oci/load_balancer_spec.go, where backends are rebuilt from the remaining nodes, and the NLB client code that drops IsDrain. The proposed change keeps a backend whose Node still has the ToBeDeletedByClusterAutoscaler taint but sets isDrain: true, and removes it once the Node is gone. Done means the opt-in annotation works and a test shows the drained backend is restored when the taint is removed. Confirm the approach with maintainers before coding.

Written by the indexing model from the issue text.

Description

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT (with a proposed fix)

Versions

CCM Version: v1.32.2 (also seen on v1.31.2; behavior is the same on master at 93969b5)

Environment:

  • Kubernetes version: v1.32.13
  • Others: self-managed clusters on OCI (Cluster API), cluster-autoscaler v1.31.5 (clusterapi provider), OCI Network Load Balancers with externalTrafficPolicy: Local

What happened?

When cluster-autoscaler scales down a node, it first adds the ToBeDeletedByClusterAutoscaler taint and then evicts pods. The generic service controller in k8s.io/cloud-provider excludes tainted nodes from every LoadBalancer Service (stableNodeSetPredicates / nodeUnTaintedPredicate). On the next sync of the Service, the OCI CCM rebuilds the backend set from the remaining nodes (getBackends) and the update deletes the tainted node's backend. OCI removes the backend immediately, and established connections through that node stop receiving packets with no reset. Clients hang until their own timeouts.

This happens while the node and its pods are still running. In our clusters the CCM typically removes the backend within seconds of the taint, because other node churn triggers a sync, while cluster-autoscaler is still evicting pods for another minute or two. Clients holding keep-alive connections through that node (for example HTTP connection pools to an ingress controller) see requests hang until their timeouts, and the stuck connections back up their pools. On busy clusters this happens on most scale-downs of a node carrying load balancer traffic.

The k8s code acknowledges the gap: the comment on stableNodeSetPredicates says tainted nodes are excluded on the next sync "and cause connections on said node to not connection drain". It was discussed upstream as a provider concern (KEP-3458, KEP-3836, and kubernetes/kubernetes#129605, where SIG Cloud Provider had no generic fix).

For comparison, the AWS cloud controller manager leaves draining to the load balancer: for NLBs it deregisters targets (ensureTargetGroupTargets), AWS keeps existing flows during the target group's deregistration delay (300s by default), and the CCM treats targets in DeregistrationInProgress as already gone (L930). For Classic ELBs it exposes service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled and -connection-draining-timeout (L130-L134). OCI backends support the same idea through isDrain ("stops receiving new connections but continues to receive traffic on existing connections"), but the CCM never sets it: removed backends are always deleted outright.

We also confirmed that an unhealthy NLB backend keeps forwarding established connections; only removing the backend cuts them. So the health check failing early (KEP-3836) is not the problem; the deletion is.

What you expected to happen?

A node that is being scaled down keeps its existing load balancer connections until the node is actually gone, the same as on AWS: the backend is drained (no new connections) instead of deleted, and removed once the node no longer exists.

How to reproduce it (as minimally and precisely as possible)?

  1. Create a type: LoadBalancer Service (NLB, externalTrafficPolicy: Local or Cluster) and hold a long-lived TCP connection through one backend node (e.g. HTTP keep-alive sending a request every second).
  2. Add the cluster-autoscaler taint to that node by hand: kubectl taint node <node> ToBeDeletedByClusterAutoscaler=<unix-ts>:NoSchedule
  3. Trigger a sync of the Service (any change to its annotations, or any node add/remove in the cluster).
  4. The CCM removes the node's backend and the long-lived connection hangs, even though the node and the pod behind it are still healthy.

Anything else we need to know?

Proposed fix, opt-in per Service so default behavior doesn't change:

  • New annotations oci-network-load-balancer.oraclecloud.com/drain-backends-on-scale-down: "true" (NLB) and service.beta.kubernetes.io/oci-load-balancer-drain-backends-on-scale-down: "true" (LB).
  • When a backend is missing from the desired set but its Node still exists with the ToBeDeletedByClusterAutoscaler taint, keep the existing backend unchanged except isDrain: true. If the autoscaler aborts the scale-down and removes the taint, the backend is un-drained. Any other missing backend (Node deleted, exclude label, node selector change) is removed as today.
  • No timers or extra state: the drain decision is recomputed from the Node object on every sync, using the CCM's existing node lister. The NLB client also needs to read and send IsDrain, which it currently drops.

If you agree this should be fixed and are able to prioritize reviewing it, I'm willing to open a PR with this approach and sign the OCA.

Dominant language
Go
Stars
160
Forks
109
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oracle/oci-cloud-controller-manager

All issues in oracle/oci-cloud-controller-manager

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.