kubernetes/ingress-nginx

Enabling Helm .Values.controller.opentelemetry.enabled prevents pods from being rescheduled on different nodes and cluster auto-scaling from working.

Open

#10,911 opened on Jan 24, 2024

View on GitHub
 (14 comments) (0 reactions) (1 assignee)Go (15,199 stars) (7,888 forks)batch import
area/docshelp wantedkind/documentationneeds-priorityneeds-triage

Description

What happened: Enabling .Values.controller.opentelemetry.enabled prevents pods from being rescheduled on different nodes and cluster auto-scaling from working.

What you expected to happen: Expected cluster auto-scaling to continue to reschedule ingres-nginx pods on different nodes

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): registry.k8s.io/ingress-nginx/controller:v1.9.5@sha256:b3aba22b1da80e7acfc52b115cae1d4c687172cbf2b742d5b502419c25ff340e

The helm chart adds emptyDir: {} here when the .Values.controller.opentelemetry.enabled is set to true: https://github.com/kubernetes/ingress-nginx/blob/0c3d52bade1219c1d61d486e44a91ded48766aac/charts/ingress-nginx/templates/controller-daemonset.yaml#L215

I think it should also add the annotation in podAnnotations area in the same file... cluster-autoscaler.kubernetes.io/safe-to-evict: "true" automatically in the helm chart when .Values.controller.opentelemetry.enabled or at least give a warning that you might want to consider adding it yourself because cluster autoscaling won't be able to move your ingress-nginx pods otherwise.

Contributor guide

Tech stack
go
Domain
devopsinfrastructureobservability
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Helm chart understandingKubernetes pod annotationscluster autoscaler basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The issue concerns the Helm chart in charts/ingress nginx/templates/controller daemonset.yaml where enabling OpenTelemetry adds an emptyDir but does not add a safe to evict annotation. The fix should add 'cluster autoscaler.kubernetes.io/safe to evict: "true"' in the podAnnotations section conditionally. Review the existing chart logic and ensure compatibility with other configurations. Consider also checking if the same issue exists for the deployment variant.
Enabling Helm .Values.controller.opentelemetry.enabled prevents pods from being rescheduled on different nodes and cluster auto-scaling from working. · kubernetes/ingress-nginx#10911 | Good First Issue