Surface the Outdated phase, and tell a runner version deprecation apart from a rejected spec

Aperta
#4,739 1 commento 4 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github-actions, go, kubernetes, prometheus

Direzione di ricerca

Start by tracing how the controller handles runner exit code 7 and updates AutoscalingRunnerSet status to Outdated; inspect the related changes in #4652 and #4660. Then find where controller metrics and Kubernetes Events are registered, and how runner version information is available. Done means Outdated sets emit a Warning Event and phase metric, while version deprecation is recorded distinctly in status.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happens

When GitHub deprecates a runner version, the runner exits 7 and the AutoscalingRunnerSet goes
Outdated. On 0.14.1 that deleted the scale set. On 0.15.x (#4652, #4660) it is switched off
until the runner spec changes, which is better. Two gaps remain:

  1. Nothing signals it. The controller records no Kubernetes Event, exports no metric for the
    phase, and the listener (the source of the gha_* metrics) is stopped. Jobs for the label
    queue without any error. We found two outages only when users reported stuck jobs, days later.
  2. A version deprecation looks the same as a rejected spec. The recovery differs: for a
    deprecation, the fix is a newer actions-runner image, and the spec was valid the day before.
Request
  • Record a Warning Event on the AutoscalingRunnerSet when it enters Outdated, and expose its
    phase as a controller metric (for example gha_autoscaling_runner_set_phase{phase}).
  • Carry the reason in status, for example a condition with reason RunnerVersionDeprecated and
    the runner version, when exit code 7 comes from a version deprecation.
Workaround

kube-state-metrics custom resource metrics on status.phase, plus an alert on
phase != Running for 10 minutes.

Controller and chart 0.14.1. Behavior checked against the 0.15.1 source.

Lingua principale
Go
Stelle
6.7k
Fork
1.5k
Merge medio
1h 34m
PR unite (30g)
5

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di actions/actions-runner-controller

Tutte le issue di actions/actions-runner-controller

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.