Node lookups fail for every OCI node when the cluster has a node another provider owns (e.g. KWOK)

Aperta
  • Qualcosa non va?
#556 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
62/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go
Ambito
cloud

Direzione di ricerca

The two node-list loops that return early on an unparseable provider ID are getCompartmentIDByInstanceID and getNodeIpFamily in pkg/cloudprovider/providers/oci/instances.go, and extractNodeAddresses calls the latter. Start by reproducing with the proposed TestLookupsSkipNodesThatAreNotOCIInstances, which should fail with the provider-ID error on current code. Done means those walks skip non-OCI nodes, the new isOCIInstanceNode check passes its tests, and the existing tests still pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

When a cluster contains a node whose spec.providerID is not an OCI one, such as a KWOK simulated node (kwok:<name>) or a node that has no provider ID yet, the cloud controller can fail to initialize real OCI workers. They stay tainted node.cloudprovider.kubernetes.io/uninitialized indefinitely.

Separately, every such node is looked up in OCI by name on every node-status cycle. With many of them this hits the OCI API rate limit.

Versions

v1.36.0, v1.35.1, v1.34.3; master has the same code.

Cause

getCompartmentIDByInstanceID and getNodeIpFamily (pkg/cloudprovider/providers/oci/instances.go) iterate over every node in the lister. They return an error at the first node whose provider ID MapProviderIDToResourceID cannot parse, before checking whether that node is the one being looked up:

for _, node := range nodeList {
    providerID, err := MapProviderIDToResourceID(node.Spec.ProviderID)
    if err != nil {
        return "", errors.New("Failed to map providerID to instanceID")
    }
    if providerID == instanceID {

getNodeIpFamily has no cache and is called from extractNodeAddresses for every node being initialized. So whenever any unrelated node is listed before the target, initialization fails with:

failed to get instance metadata for node <real-worker>: error fetching node by provider ID: Failed to map providerID to instanceID, and error by node name: Failed to map providerID to instanceID

The lister's order is unpredictable, so with many such nodes initialization almost never succeeds.

The extra OCI traffic comes from two places:

  • Kubernetes' cloud node controller falls back to NodeAddresses(nodeName) when NodeAddressesByProviderID fails, and that calls GetInstanceByNodeName.
  • The node-info controller calls InstanceID(nodeName) for nodes with no provider ID.
Observed

A CAPI (CAPOCI) cluster on Kubernetes v1.36.6 with 1,322 KWOK nodes and about 10 OCI workers:

  • Newly provisioned workers joined (Ready) but never received an InternalIP or OCI labels.
  • The controller logged hundreds of GetPrimaryVNICForInstance: rate limited(read) for operation: ListVnicAttachments errors per node-status cycle.
Fix

Proposed fix, PR to follow (patch against v1.36.0; it applies unchanged to v1.35.1 and v1.34.3):

  • The two node walks skip nodes whose provider ID is not OCI's (continue instead of return).
  • isOCIInstanceNode: a node is not an OCI instance if its provider ID is another provider's, or if KWOK marks it (kwok.x-k8s.io/node: fake, as a label or annotation; KWOK can simulate nodes without a provider ID). A node without a provider ID is otherwise still treated as a joining OCI worker.
  • The node-info controller and NodeAddresses do not look up such nodes in OCI.
  • Regression tests:
    • TestLookupsSkipNodesThatAreNotOCIInstances fails on the current code with the error above;
    • TestIsOCIInstanceNode and TestNodeAddressesOfANodeThatIsNotAnOCIInstance cover the new check.

With the patched controller on the cluster above, every worker initialized, and over full node-status cycles there were no provider-ID failures and no rate limiting.

Lingua principale
Go
Stelle
160
Fork
109
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di oracle/oci-cloud-controller-manager

Tutte le issue di oracle/oci-cloud-controller-manager

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.