Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Mitigation eval passes when the broken workload is deleted or scaled to zero

Aperta
  • Qualcosa non va?
#208 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

@TheDeadcoder ci sta già lavorando.

Dal 12/10/2026.

  • #209 di @TheDeadcoder — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kubernetes, python
Ambito
backend, testing

Direzione di ricerca

Start by reading MitigationTask.wait_until_pods_healthy, pods_unhealthy_reasons, and KubeCtl._pod_is_ready_or_succeeded, then compare the four problems that use their own pod checks. The reproduction uses revoke_auth_mongodb-mitigation-1 and a kind cluster; first verify the reported scale-to-zero case. Done means eval fails when a required workload is missing or scaled to zero, and the listed false-success cases are covered.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happens

Most mitigation problems use MitigationTask.wait_until_pods_healthy to decide if a fix worked. This check only looks at pods that exist. It also skips pods that have no container statuses.

Because of this, eval can report success while the service is still broken. I found four ways this happens:

  1. The broken workload is deleted or scaled to 0. Its crashing pods disappear. All the pods that are left look healthy. Eval passes.
  2. A healthy workload is scaled to 0. Its pods disappear too. Eval passes.
  3. A pod cannot be scheduled. It stays Pending and has no container statuses. The check skips it. So assign_to_non_existent_node_social_net-mitigation-1 passes even when the agent does nothing.
  4. The fault is a feature flag. astronomy_shop_kafka_queue_problems-mitigation-1 turns on a flagd feature flag. Every pod stays healthy. So eval passes even when the agent does nothing.

How to reproduce

I tested this on main (commit ccf08d0) with a kind cluster.

  1. Start revoke_auth_mongodb-mitigation-1. The geo pods start crash-looping because MongoDB rejects their credentials.
  2. Run kubectl scale deployment geo -n hotel-reservation --replicas=0.
  3. Call submit().
  4. Eval returns success once the geo pods are gone. But every request that needs geo still fails.

What I measured

I ran all 14 mitigation problems once each. After each change, I polled eval for 120 seconds. This is the same window AIOpsLab uses by default. I also checked the cluster myself: ready replicas, Service endpoints, and wrk2 requests.

What I did after the fault How often eval passed
Scaled the broken Deployment to 0 5 of 6 problems
Deleted the broken Deployment 5 of 6 problems
Ran recover_fault(), then scaled a healthy Deployment to 0 13 of 13 problems
Did nothing and polled eval for 300 seconds 3 of 14 problems passed the whole time

Notes on the table:

  • The first two rows apply only to the 6 problems that leave a Deployment unhealthy after the fault.
  • The 3 problems that passed with no action are assign_to_non_existent_node_social_net, astronomy_shop_kafka_queue_problems and redeploy_without_PV.
  • misconfig_app_hotel_res also passed in its first few polls with no action.
  • In every pass from the first three rows, my own check showed the removed workload had no ready replicas. If the workload had a Service, that Service had no endpoints.

Why it happens

  1. pods_unhealthy_reasons skips any pod with empty container_statuses. But KubeCtl._pod_is_ready_or_succeeded treats the same pod as not ready. The two checks disagree.
  2. Nothing records which workloads the app deployed. So when a workload is deleted or scaled to 0, the pod list shows nothing wrong.
  3. Four problems use their own pod checks instead: misconfig_app_hotel_res, wrong_bin_usage, redeploy_without_pv and scale_pod_social_net.
    • They also miss a workload that was deleted or scaled to 0. Two of them check one workload by name, but no others.
    • They do not use the settle window from #180.
    • Three of them loop over pod.status.container_statuses without checking for None. A Pending pod makes them raise a TypeError.

Suggested fix

I will open a PR with these changes:

  1. Treat a pod with no container statuses as unhealthy, unless it has finished successfully.
  2. Record the Deployments and StatefulSets right after the app is deployed, before the fault is injected. Then fail if a recorded workload is gone or scaled to 0.
  3. Make the four problems above use the shared check.
  4. In the kafka problem, also check that the kafkaQueueProblems flag is off again.

Not covered by this fix

  1. redeploy_without_PV: the pods come back healthy without their volumes. Pod state cannot show this data loss.
  2. misconfig_app_hotel_res: eval can pass before the crash shows up. Fixing this needs the app to stay healthy for a while, not just once.
Lingua principale
Python
Stelle
1k
Fork
177
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/AIOpsLab

Tutte le issue di microsoft/AIOpsLab

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.