grafana/loki

add rule option to set alertmanager endsAt time

Open

#7,450 opened on Oct 18, 2022

View on GitHub
 (8 comments) (0 reactions) (1 assignee)Go (3,997 forks)batch import
component/rulergood first issuetype/feature

Repository metrics

Stars
 (28,187 stars)
PR merge metrics
 (Avg merge 5d 20h) (522 merged PRs in 30d)

Description

Is your feature request related to a problem? Please describe. Currently there is no option to set the resolve timeout (endsAt) value in an alerting rule, I want to set the endsAt value for cases when some log event happens once and I want to keep the alert in alertmanager for a long time and not get resolved based on the global alertmanager resolve_timeout .

Describe the solution you'd like

add an endsAt ( time from now) parameter in the rule, that will be passed to alertmanager

  - name: should_fire
    rules:
      - alert: HighPercentageError
        endsAt: 1d
        expr: |
          sum(rate({app="foo", env="production"} |= "error" [5m])) by (job)
            /
          sum(rate({app="foo", env="production"}[5m])) by (job)
            > 0.05
        for: 10m
        labels:
            severity: page
        annotations:
            summary: High request latency```

**Describe alternatives you've considered**
-
**Additional context**
-

Contributor guide

Research direction
Investigate how alerting rules are defined and parsed in Loki, particularly the rule model in pkg/ruler. Identify where the endsAt field would be added in the rule type and how it maps to Alertmanager's endsAt in the alerts sent. Follow the code path from rule evaluation to alert dispatch in the ruler package.
Tech stack
go
Domain
backendapiinfrastructure
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Goalerting conceptsLoki codebase structure
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40