Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Mitigation eval passes when the broken workload is deleted or scaled to zero

Đang mở
  • Có gì không ổn?
#208 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

@TheDeadcoder đang làm issue này rồi.

Từ ngày 12/10/2026.

  • #209 của @TheDeadcoder — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
kubernetes, python
Lĩnh vực
backend, testing

Hướng nghiên cứu

Start by reading MitigationTask.wait_until_pods_healthy, pods_unhealthy_reasons, and KubeCtl._pod_is_ready_or_succeeded, then compare the four problems that use their own pod checks. The reproduction uses revoke_auth_mongodb-mitigation-1 and a kind cluster; first verify the reported scale-to-zero case. Done means eval fails when a required workload is missing or scaled to zero, and the listed false-success cases are covered.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

What happens

Most mitigation problems use MitigationTask.wait_until_pods_healthy to decide if a fix worked. This check only looks at pods that exist. It also skips pods that have no container statuses.

Because of this, eval can report success while the service is still broken. I found four ways this happens:

  1. The broken workload is deleted or scaled to 0. Its crashing pods disappear. All the pods that are left look healthy. Eval passes.
  2. A healthy workload is scaled to 0. Its pods disappear too. Eval passes.
  3. A pod cannot be scheduled. It stays Pending and has no container statuses. The check skips it. So assign_to_non_existent_node_social_net-mitigation-1 passes even when the agent does nothing.
  4. The fault is a feature flag. astronomy_shop_kafka_queue_problems-mitigation-1 turns on a flagd feature flag. Every pod stays healthy. So eval passes even when the agent does nothing.

How to reproduce

I tested this on main (commit ccf08d0) with a kind cluster.

  1. Start revoke_auth_mongodb-mitigation-1. The geo pods start crash-looping because MongoDB rejects their credentials.
  2. Run kubectl scale deployment geo -n hotel-reservation --replicas=0.
  3. Call submit().
  4. Eval returns success once the geo pods are gone. But every request that needs geo still fails.

What I measured

I ran all 14 mitigation problems once each. After each change, I polled eval for 120 seconds. This is the same window AIOpsLab uses by default. I also checked the cluster myself: ready replicas, Service endpoints, and wrk2 requests.

What I did after the fault How often eval passed
Scaled the broken Deployment to 0 5 of 6 problems
Deleted the broken Deployment 5 of 6 problems
Ran recover_fault(), then scaled a healthy Deployment to 0 13 of 13 problems
Did nothing and polled eval for 300 seconds 3 of 14 problems passed the whole time

Notes on the table:

  • The first two rows apply only to the 6 problems that leave a Deployment unhealthy after the fault.
  • The 3 problems that passed with no action are assign_to_non_existent_node_social_net, astronomy_shop_kafka_queue_problems and redeploy_without_PV.
  • misconfig_app_hotel_res also passed in its first few polls with no action.
  • In every pass from the first three rows, my own check showed the removed workload had no ready replicas. If the workload had a Service, that Service had no endpoints.

Why it happens

  1. pods_unhealthy_reasons skips any pod with empty container_statuses. But KubeCtl._pod_is_ready_or_succeeded treats the same pod as not ready. The two checks disagree.
  2. Nothing records which workloads the app deployed. So when a workload is deleted or scaled to 0, the pod list shows nothing wrong.
  3. Four problems use their own pod checks instead: misconfig_app_hotel_res, wrong_bin_usage, redeploy_without_pv and scale_pod_social_net.
    • They also miss a workload that was deleted or scaled to 0. Two of them check one workload by name, but no others.
    • They do not use the settle window from #180.
    • Three of them loop over pod.status.container_statuses without checking for None. A Pending pod makes them raise a TypeError.

Suggested fix

I will open a PR with these changes:

  1. Treat a pod with no container statuses as unhealthy, unless it has finished successfully.
  2. Record the Deployments and StatefulSets right after the app is deployed, before the fault is injected. Then fail if a recorded workload is gone or scaled to 0.
  3. Make the four problems above use the shared check.
  4. In the kafka problem, also check that the kafkaQueueProblems flag is off again.

Not covered by this fix

  1. redeploy_without_PV: the pods come back healthy without their volumes. Pod state cannot show this data loss.
  2. misconfig_app_hotel_res: eval can pass before the crash shows up. Fixing this needs the app to stay healthy for a while, not just once.
Ngôn ngữ chính
Python
Star
1k
Fork
177
Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/AIOpsLab

Tất cả issue của microsoft/AIOpsLab

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.