kubernetes-sigs/cluster-api

Stale `HealthCheckSucceeded=False` condition not cleared after node recovery when using external remediation

Open

#13,822 opened on Jun 18, 2026

View on GitHub
 (6 comments) (0 reactions) (1 assignee)Go (1,532 forks)auto 404
area/machinehealthcheckhelp wantedkind/bugpriority/important-soontriage/accepted

Repository metrics

Stars
 (4,267 stars)
PR merge metrics
 (PR metrics pending)

Description

We have encountered an issue where machines end up with a stale HealthCheckSucceeded=False condition that never gets cleared, even though the nodes have fully recovered and are running fine.

This has been observed in two scenarios, both using a MachineHealthCheck with a remediationTemplate:

  1. "No heartbeat for 5 minutes": Node briefly reported Ready=Unknown, the health check fired, the node recovered on its own, but HealthCheckSucceeded=False persists.
  2. "Exceeded start-up time of 30m": Node eventually came up healthy after exceeding the configured startup timeout, but the failed condition was never removed.

After a node recovers from such a transient health check failure:

  • NodeHealthy=True
  • NodeReady=True
  • No active remediation objects exist
  • HealthCheckSucceeded=False: this was not cleared
  • Available=False: machine appears unavailable when it is not

Label(s) to be applied

/kind bug /area machinehealthcheck

Contributor guide

Research direction
Investigate the condition update logic in the MachineHealthCheck controller, specifically how `HealthCheckSucceeded` and `Available` conditions are managed after node recovery when remediation is external. Focus on the condition reconciliation that may not clear stale failure conditions.
Tech stack
go
Domain
backend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Go
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65