[AppSec] Unbounded memory growth under load : coraza transformation-cache bug (fixed upstream by corazawaf/coraza#1682, please pull it into the crowdsec coraza fork)

オープン
  • 何か問題がありますか？
#4,797 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
25/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
go
領域
performance, security

調査の方向性

The fix is upstream in corazawaf/coraza#1682, so the work is pinning a newer coraza fork version. Start with the replace directive for coraza in the crowdsec go.mod, and check whether the crowdsecurity/coraza fork can take the merged change. Done means the fork pin moves past the 2026-10-15 merge and the AppSec heap stays flat under the reproduction load, which needs a maintainer to review.

索引モデルが issue の本文から書いたものです。

説明

kind/bug needs/triage os/linux
What happened?

🤖 Note: AI Generated Issue with Human in the loop.

On v1.8.2, AppSec memory grows without bound under rule-evaluation load until the process is OOM-killed. On a 4 GB host (no swap) it took the whole machine down.

Setup: AppSec in outofband with the CRS ruleset, two fronts behind a load balancer. It was triggered by a burst of scanner traffic (WordPress-vuln probing) while one front briefly carried 100% of the traffic after an LB failover.

Observed:

  • Front A: crowdsec RSS ~2.5 GB → saturated the VM, host unreachable, hard reboot.
  • Front B (~1h50 in): HeapAlloc ≈ 1.6 GB, Sys ≈ 2.88 GB, RSS ≈ 2.07 GB — live Go heap, not off-heap.
  • At idle/normal load it stays ~90 MB; growth tracks AppSec rule-evaluation volume.

The heap profile (inuse_space) is dominated by coraza (*Rule).transformArg:

14.2 MB  github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg   rule.go:482
11.5 MB  github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg   rule.go:482
 2.7 MB  github.com/petar-dambovaliev/aho-corasick.(*iNFA).addSparseState         nfa.go:177
 1.1 MB  github.com/corazawaf/coraza/v3/internal/corazawaf.(*RuleGroup).Add       rulegroup.go:64

transformArg aggregates to ~73% of live heap even on a light 33 MB snapshot. The goroutine profile is normal (179 total), so this is not a goroutine / channel / plugin-broker leak.

This matches corazawaf/coraza#1681, already confirmed as a CrowdSec-AppSec OOM cause on that thread and fixed by corazawaf/coraza#1682. CrowdSec 1.8.2 pins the crowdsec coraza fork v3.8.0-crowdsec.20261001 (snapshot 2026-10-01), which predates the fix (merged 2026-10-15). v1.7.8 pinned an older fork (v3.3.3-crowdsec) and did not show this.

What did you expect to happen?

AppSec memory to stay bounded under sustained rule-evaluation load (as on v1.7.8), instead of growing until OOM.

How can we reproduce it (as minimally and precisely as possible)?
  1. CrowdSec v1.8.2, AppSec enabled with a CRS appsec-config, in outofband.
  2. A memory-constrained host (or a systemd MemoryMax on the service) so the OOM is visible.
  3. Sustained traffic hitting many rules against map-backed arg collections (ARGS / ARGS_NAMES) — e.g. a WordPress-vuln scanner probing many paths with query/body args.
  4. curl 127.0.0.1:6060/debug/pprof/heap (inuse_space) grows with (*Rule).transformArg dominant and RSS climbs until OOM. Idle / low traffic does not reproduce it.
Anything else we need to know?

The root cause is upstream and understood: corazawaf/coraza#1681. The per-phase transformation cache is keyed by an argument's positional index, but map-backed collections (ARGS, headers, cookies) iterate in random order, so the same argument lands at a different index in each rule. That causes cache misses, re-runs the transformation chain, and stores duplicate entries (23–34× the needed entries).

It reaches gigabytes because Go maps never shrink and transactions are reused from a per-P sync.Pool: each pooled transaction keeps its high-water capacity for the life of the process, and many-argument payloads compound it across routines. corazawaf/coraza#1681 only reproduced modest per-transaction retention (~1–6 MiB) and noted it could not reproduce the GB-scale heap of corazawaf/coraza#1680 — our ~1.6 GB is that missing production data point.

corazawaf/coraza#1682 fixes it (merged 2026-10-15): it keys cache entries by value identity and clears the cache in Transaction.Close(). It is not in a tagged coraza release yet (latest v3.8.1, 2026-10-02), and 1.8.2 pins the fork v3.8.0-crowdsec.20261001 (2026-10-01) — both from before the merge. Pulling #1682 into the crowdsecurity/coraza fork and bumping the replace would resolve it.

Until then, the hosts stay up by capping crowdsec's RAM (systemd MemoryMax) and by emptying the AppSec outofband_rules, which stops rule evaluation so the growth stops at the source.

Crowdsec version
$ cscli version
version: v1.8.2-debian-pragmatic-amd64-3d369d64
Codename: alphaga
BuildDate: 2026-10-27_14:05:00
GoVersion: 1.27.1
Platform: linux
libre2: C++
User-Agent: crowdsec/v1.8.2-debian-pragmatic-amd64-3d369d64-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Enabled collections and parsers
$ cscli hub list -o raw
# paste output here
Acquisition config
# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
# paste output here

# On Windows:
C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
# paste output here
Config show
$ cscli config show
# paste output here
Prometheus metrics
$ cscli metrics
# paste output here
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
主要言語
Go
スター
15.1k
フォーク
726
平均マージ
2日 51分
マージ済み PR（30日）
18

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

crowdsecurity/crowdsec のほかの issue

crowdsecurity/crowdsec の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。