[AppSec] Unbounded memory growth under load : coraza transformation-cache bug (fixed upstream by corazawaf/coraza#1682, please pull it into the crowdsec coraza fork)

Aperta
  • Qualcosa non va?
#4,797 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
performance, security

Direzione di ricerca

The fix is upstream in corazawaf/coraza#1682, so the work is pinning a newer coraza fork version. Start with the replace directive for coraza in the crowdsec go.mod, and check whether the crowdsecurity/coraza fork can take the merged change. Done means the fork pin moves past the 2026-10-15 merge and the AppSec heap stays flat under the reproduction load, which needs a maintainer to review.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

kind/bug needs/triage os/linux
What happened?

🤖 Note: AI Generated Issue with Human in the loop.

On v1.8.2, AppSec memory grows without bound under rule-evaluation load until the process is OOM-killed. On a 4 GB host (no swap) it took the whole machine down.

Setup: AppSec in outofband with the CRS ruleset, two fronts behind a load balancer. It was triggered by a burst of scanner traffic (WordPress-vuln probing) while one front briefly carried 100% of the traffic after an LB failover.

Observed:

  • Front A: crowdsec RSS ~2.5 GB → saturated the VM, host unreachable, hard reboot.
  • Front B (~1h50 in): HeapAlloc ≈ 1.6 GB, Sys ≈ 2.88 GB, RSS ≈ 2.07 GB — live Go heap, not off-heap.
  • At idle/normal load it stays ~90 MB; growth tracks AppSec rule-evaluation volume.

The heap profile (inuse_space) is dominated by coraza (*Rule).transformArg:

14.2 MB  github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg   rule.go:482
11.5 MB  github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg   rule.go:482
 2.7 MB  github.com/petar-dambovaliev/aho-corasick.(*iNFA).addSparseState         nfa.go:177
 1.1 MB  github.com/corazawaf/coraza/v3/internal/corazawaf.(*RuleGroup).Add       rulegroup.go:64

transformArg aggregates to ~73% of live heap even on a light 33 MB snapshot. The goroutine profile is normal (179 total), so this is not a goroutine / channel / plugin-broker leak.

This matches corazawaf/coraza#1681, already confirmed as a CrowdSec-AppSec OOM cause on that thread and fixed by corazawaf/coraza#1682. CrowdSec 1.8.2 pins the crowdsec coraza fork v3.8.0-crowdsec.20261001 (snapshot 2026-10-01), which predates the fix (merged 2026-10-15). v1.7.8 pinned an older fork (v3.3.3-crowdsec) and did not show this.

What did you expect to happen?

AppSec memory to stay bounded under sustained rule-evaluation load (as on v1.7.8), instead of growing until OOM.

How can we reproduce it (as minimally and precisely as possible)?
  1. CrowdSec v1.8.2, AppSec enabled with a CRS appsec-config, in outofband.
  2. A memory-constrained host (or a systemd MemoryMax on the service) so the OOM is visible.
  3. Sustained traffic hitting many rules against map-backed arg collections (ARGS / ARGS_NAMES) — e.g. a WordPress-vuln scanner probing many paths with query/body args.
  4. curl 127.0.0.1:6060/debug/pprof/heap (inuse_space) grows with (*Rule).transformArg dominant and RSS climbs until OOM. Idle / low traffic does not reproduce it.
Anything else we need to know?

The root cause is upstream and understood: corazawaf/coraza#1681. The per-phase transformation cache is keyed by an argument's positional index, but map-backed collections (ARGS, headers, cookies) iterate in random order, so the same argument lands at a different index in each rule. That causes cache misses, re-runs the transformation chain, and stores duplicate entries (23–34× the needed entries).

It reaches gigabytes because Go maps never shrink and transactions are reused from a per-P sync.Pool: each pooled transaction keeps its high-water capacity for the life of the process, and many-argument payloads compound it across routines. corazawaf/coraza#1681 only reproduced modest per-transaction retention (~1–6 MiB) and noted it could not reproduce the GB-scale heap of corazawaf/coraza#1680 — our ~1.6 GB is that missing production data point.

corazawaf/coraza#1682 fixes it (merged 2026-10-15): it keys cache entries by value identity and clears the cache in Transaction.Close(). It is not in a tagged coraza release yet (latest v3.8.1, 2026-10-02), and 1.8.2 pins the fork v3.8.0-crowdsec.20261001 (2026-10-01) — both from before the merge. Pulling #1682 into the crowdsecurity/coraza fork and bumping the replace would resolve it.

Until then, the hosts stay up by capping crowdsec's RAM (systemd MemoryMax) and by emptying the AppSec outofband_rules, which stops rule evaluation so the growth stops at the source.

Crowdsec version
$ cscli version
version: v1.8.2-debian-pragmatic-amd64-3d369d64
Codename: alphaga
BuildDate: 2026-10-27_14:05:00
GoVersion: 1.27.1
Platform: linux
libre2: C++
User-Agent: crowdsec/v1.8.2-debian-pragmatic-amd64-3d369d64-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Enabled collections and parsers
$ cscli hub list -o raw
# paste output here
Acquisition config
# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
# paste output here

# On Windows:
C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
# paste output here
Config show
$ cscli config show
# paste output here
Prometheus metrics
$ cscli metrics
# paste output here
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
Lingua principale
Go
Stelle
15.1k
Fork
726
Merge medio
2g 51m
PR unite (30g)
18

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di crowdsecurity/crowdsec

Tutte le issue di crowdsecurity/crowdsec

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.