Retention for expired decisions, independent of alert retention

Aperta
  • Qualcosa non va?
#4,703 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
databases

Direzione di ricerca

Start by reading the existing flush.max_age behavior and FlushOrphans, then review issue #4613 for measurements of the longestDecisionForScopeTypeValue query shape. The change should allow expired decisions to be retained independently of their parent alerts while preserving the configured window for reporting deletions to bouncers. Verify the new behavior and configuration with tests, including the stated trade-off for inspecting old alerts.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

kind/enhancement needs/triage

/kind enhancement

What would you like to be added?

A retention setting for expired decisions that's independent of alert retention — something like db_config.flush.decisions_max_age. Expired decision rows would be deleted that long after they expire, separately from their parent alert, much as FlushOrphans already deletes ownerless expired decisions.

Why is this needed?

flush.max_age is currently doing three unrelated jobs:

  1. how long alerts stay queryable
  2. how much dead weight every stream query walks through
  3. how long an expired decision stays reportable to bouncers in the deleted array

There's no way to tune them separately, and 1 and 3 pull against each other.

On our LAPI (~1.2k bouncers, 60s poll) decisions sits at 612k rows with 301k already expired: a 6-hourly blocklist import with an 8h TTL against max_age: 26h leaves each batch dead in the table for 18h. Every startup=true resync walks all of it through the longestDecisionForScopeTypeValue anti-join — #4613 has measurements for that query shape.

Pushing the ban TTL past max_age fixes the bloat, since #4527 keeps an alert alive while its decisions are active. But then the alert is flushed within a minute of the last decision expiring and the cascade hard-deletes the rows, so a bouncer only learns about the unban if it polls inside that ~60s window. Roughly half the fleet misses it.

What I'd like to be able to configure: 7d of alert history, deletions reported for 12h, and not carrying 7d of expired decisions in every stream query.

The obvious cost is that cscli alerts inspect on an old alert would show fewer decisions than it does today. That seems like the right trade to me, but it's a behaviour change worth naming.

Lingua principale
Go
Stelle
15.1k
Fork
726
Merge medio
2g 51m
PR unite (30g)
18

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di crowdsecurity/crowdsec

Tutte le issue di crowdsecurity/crowdsec

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.