Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Mitigation eval passes when the broken workload is deleted or scaled to zero

Abierto
  • ¿Algo no está bien?
#208 0 comentarios 0 reacciones 0 asignados Ver en GitHub

@TheDeadcoder ya está trabajando en esto.

Desde el 12/10/2026.

  • #209 de @TheDeadcoder — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
kubernetes, python
Área
backend, testing

Línea de trabajo

Start by reading MitigationTask.wait_until_pods_healthy, pods_unhealthy_reasons, and KubeCtl._pod_is_ready_or_succeeded, then compare the four problems that use their own pod checks. The reproduction uses revoke_auth_mongodb-mitigation-1 and a kind cluster; first verify the reported scale-to-zero case. Done means eval fails when a required workload is missing or scaled to zero, and the listed false-success cases are covered.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What happens

Most mitigation problems use MitigationTask.wait_until_pods_healthy to decide if a fix worked. This check only looks at pods that exist. It also skips pods that have no container statuses.

Because of this, eval can report success while the service is still broken. I found four ways this happens:

  1. The broken workload is deleted or scaled to 0. Its crashing pods disappear. All the pods that are left look healthy. Eval passes.
  2. A healthy workload is scaled to 0. Its pods disappear too. Eval passes.
  3. A pod cannot be scheduled. It stays Pending and has no container statuses. The check skips it. So assign_to_non_existent_node_social_net-mitigation-1 passes even when the agent does nothing.
  4. The fault is a feature flag. astronomy_shop_kafka_queue_problems-mitigation-1 turns on a flagd feature flag. Every pod stays healthy. So eval passes even when the agent does nothing.

How to reproduce

I tested this on main (commit ccf08d0) with a kind cluster.

  1. Start revoke_auth_mongodb-mitigation-1. The geo pods start crash-looping because MongoDB rejects their credentials.
  2. Run kubectl scale deployment geo -n hotel-reservation --replicas=0.
  3. Call submit().
  4. Eval returns success once the geo pods are gone. But every request that needs geo still fails.

What I measured

I ran all 14 mitigation problems once each. After each change, I polled eval for 120 seconds. This is the same window AIOpsLab uses by default. I also checked the cluster myself: ready replicas, Service endpoints, and wrk2 requests.

What I did after the fault How often eval passed
Scaled the broken Deployment to 0 5 of 6 problems
Deleted the broken Deployment 5 of 6 problems
Ran recover_fault(), then scaled a healthy Deployment to 0 13 of 13 problems
Did nothing and polled eval for 300 seconds 3 of 14 problems passed the whole time

Notes on the table:

  • The first two rows apply only to the 6 problems that leave a Deployment unhealthy after the fault.
  • The 3 problems that passed with no action are assign_to_non_existent_node_social_net, astronomy_shop_kafka_queue_problems and redeploy_without_PV.
  • misconfig_app_hotel_res also passed in its first few polls with no action.
  • In every pass from the first three rows, my own check showed the removed workload had no ready replicas. If the workload had a Service, that Service had no endpoints.

Why it happens

  1. pods_unhealthy_reasons skips any pod with empty container_statuses. But KubeCtl._pod_is_ready_or_succeeded treats the same pod as not ready. The two checks disagree.
  2. Nothing records which workloads the app deployed. So when a workload is deleted or scaled to 0, the pod list shows nothing wrong.
  3. Four problems use their own pod checks instead: misconfig_app_hotel_res, wrong_bin_usage, redeploy_without_pv and scale_pod_social_net.
    • They also miss a workload that was deleted or scaled to 0. Two of them check one workload by name, but no others.
    • They do not use the settle window from #180.
    • Three of them loop over pod.status.container_statuses without checking for None. A Pending pod makes them raise a TypeError.

Suggested fix

I will open a PR with these changes:

  1. Treat a pod with no container statuses as unhealthy, unless it has finished successfully.
  2. Record the Deployments and StatefulSets right after the app is deployed, before the fault is injected. Then fail if a recorded workload is gone or scaled to 0.
  3. Make the four problems above use the shared check.
  4. In the kafka problem, also check that the kafkaQueueProblems flag is off again.

Not covered by this fix

  1. redeploy_without_PV: the pods come back healthy without their volumes. Pod state cannot show this data loss.
  2. misconfig_app_hotel_res: eval can pass before the crash shows up. Fixing this needs the app to stay healthy for a while, not just once.
Lenguaje dominante
Python
Estrellas
1k
Forks
177
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/AIOpsLab

Todos los issues de microsoft/AIOpsLab

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.