Allow configuration of partial evaluation strategies for rule group evaluation failures

Proposal

Current Behaviour

Currently, Prometheus evaluates each rule within a rule group independently. When one rule fails to be evaluated (e.g. due to a network issue to the metric storage), we continue to evaluate subsequent rules in the same rule group. See https://github.com/prometheus/prometheus/blob/c0c22ed04200a8d24d1d5719f605c85710f0d008/rules/manager.go#L605. This strategy works in many contexts, but can be very undesirable in other contexts.

Why Current Behaviour Is Undesirable

One such undesirable context is when we are calculating SLOs (which normally represents the ratio of "good events" vs "total events"). It makes sense to evaluate "good events" as one recording rule, and "total events" as a different recording rule within the same rule group. When rules are evaluated independently and there are transient network issues, it is possible that the good events get evaluated and updated while the total events failed to be evaluated (or vice versa), this makes subsequent calculations of SLOs wrong.

To illustrate with a semi-realistic scenario, think "good events" = number of requests that succeeded, "total events" = number of total requests. For t=0, all 2 requests succeeded, for t=1, all 3 requests succeeded. Imagine at t=0, evaluations were fine. At t=1, evaluation of "total events" succeeded, so "total events" = 3, evaluation of "good events" failed, so "good events" still = 2 (metric has gone stale). As a result, we would be generating false alarms, as we think there are fewer good events than total events, when in fact there weren't any bad events. This kind of false alarm scenarios can become common when there are thousands of rules and an unstable network. Note this may be less common when Prometheus is evaluating rules from local tsdb, but it is more common when evaluating rules using a remote storage.

Proposed Feature

Allow users to specify per-rule-group strategies to handle partial evaluation failures within a rule group.

Possible strategies:

• independently evaluate each rule (current behaviour)
• "when 1 rule fails, all rules fail" (desirable behaviour for SLO example above)
• any others?