Disabling Maintenance mode does not trigger an event
#11,330 opened on Oct 15, 2021
Repository metrics
- Stars
- (29,891 stars)
- PR merge metrics
- (Avg merge 8d 12h) (41 merged PRs in 30d)
Description
Overview of the Issue
I was looking at disabling alerts in consul-alerts when host is in maintenance mode: https://github.com/AcalephStorage/consul-alerts/issues/213
And I found out that while consul-alerts does create an entry in the KV store for _node_maintenance service check when maintenance mode is disabled, it does NOT update that entry once maintenance mode is disabled:

So I tried running consul watch myself same as the consul-alerts service does and I found out that disabling maintenance mode does not trigger an event:
> consul watch -type checks cat | jq '.[] | select(.CheckID == "_node_maintenance")'
{
"Node": "node-01.example.org",
"CheckID": "_node_maintenance",
"Name": "Node Maintenance Mode",
"Status": "critical",
"Notes": "TEST",
"Output": "",
"ServiceID": "",
"ServiceName": "",
"ServiceTags": [],
"Type": "maintenance",
"Definition": {
"Interval": "0s",
"Timeout": "0s",
"DeregisterCriticalServiceAfter": "0s",
"HTTP": "",
"Header": null,
"Method": "",
"Body": "",
"TLSServerName": "",
"TLSSkipVerify": false,
"TCP": ""
},
"CreateIndex": 67303625,
"ModifyIndex": 67303625
}
I receive one event with Status set to critical, but no recovery to passing, since the _node_maintenance check is removed.
For this reason consul-alerts never updates its stage of _node_maintenance check.
Reproduction Steps
- Use version
1.10.1 - Run
consul watch -type checks cat - See that no even is triggered for disabling of maintenance mode
Operating system and Environment details
Ubuntu 20.04.3, x86_64