Scheduling exceptions should trigger a cluster alarm
#366 ouverte le 20 nov. 2024
Métriques du dépôt
- Stars
- (77 étoiles)
- Métriques de merge PR
- (Merge moyen 29j 2h) (3 PRs mergées en 30 j)
Description
Short
When the root or cluster schedulers throw an exception, this must trigger an alarm in the grafana dashboard
Proposal
Ad discussed in this issue:
This solves a bug I noticed during development of this feature. If we have a malformed node, E.g. the node is broken (or malicious) and it's sending wrong info, it breaks the scheduling. So, if a node is malformed we don't consider it during scheduling and we throw an error. Then we need to extend the alarming system to catch this error and trigger an alarm (I might open an issue about this now). Rn this is simply fixing the scheduling problem and creating the error log. I discovered this because during the developent of this PR many times the node sent garbage registration info, breaking the cluster.
Originally posted by @giobart in https://github.com/oakestra/oakestra/pull/358#discussion_r1850448984
We might have cases where the scheduling fails due to malformed cluster or worker info. This incidents must be reported to the grafana dashbaord with an alarm using our Loki pipeline.
Ratio
Improved cluster alarming system
Impact
Root/Cluster loki, promtail and grafana
Development time
TBD
Status
looking for feedback
Checklist
- Discussed
- Documented
- Implemented
- Tested