Leaky bucket pour busy-wait floods crowdsec.log with thousands of 'stuck for … sending event' warnings per second (v1.8.1, follow-up to #1519)

Aperta
  • Qualcosa non va?
#4,734 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
backend, performance

Direzione di ricerca

Start in pkg/leakybucket/manager_run.go, in PourItemToBucket, and read the non-blocking select and warning logic identified in the issue. Check how the loop behaves in LIVE and TIMEMACHINE modes before choosing an approach. Reproduce with concurrent events for one bucket; done means waiting no longer burns a core or floods the log with repeated warnings.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

kind/bug needs/triage
What happened?

PourItemToBucket busy-waits while a bucket's leak routine is busy, and logs one warning every 100,000 spin iterations. When a single source IP sends a burst of events to one bucket, this floods crowdsec.log with thousands of identical warnings within the same second, and burns CPU while doing it.

This is the same tight loop reported in #1519 (2022, closed). The code is unchanged in v1.8.1. We're reopening the topic because of a side effect #1519 didn't cover: the log volume.

Observed on Windows Server 2019, CrowdSec v1.8.1 (909b5157) running as a service, mostly IIS + Windows firewall acquisition, buckets_routines > 1. In one burst on a low-traffic server, about 1.3 MB of log (roughly 5,000 lines) was written with the same timestamp (one second), all for one bucket of crowdsecurity/http-crawl-non_statics (a single crawling IP):

time="2026-10-05T02:35:13+01:00" level=warning msg="stuck for 109.3783ms sending event to ba535735750b0e3f869f3656d10399cb38cafb27 (sigclosed:0 failed_sent:12899999 attempts:12900000)" cfg=winter-butterfly name=crowdsecurity/http-crawl-non_statics
time="2026-10-05T02:35:13+01:00" level=warning msg="stuck for 140.6272ms sending event to ba535735750b0e3f869f3656d10399cb38cafb27 (sigclosed:0 failed_sent:16599999 attempts:16600000)" cfg=winter-butterfly name=crowdsecurity/http-crawl-non_statics
time="2026-10-05T02:35:13+01:00" level=warning msg="stuck for 187.5039ms sending event to ba535735750b0e3f869f3656d10399cb38cafb27 (sigclosed:0 failed_sent:23699999 attempts:23700000)" cfg=winter-butterfly name=crowdsecurity/http-crawl-non_statics
time="2026-10-05T02:35:13+01:00" level=warning msg="stuck for 106.6567ms sending event to ba535735750b0e3f869f3656d10399cb38cafb27 (sigclosed:0 failed_sent:9699999 attempts:9700000)" cfg=winter-butterfly name=crowdsecurity/http-crawl-non_statics
... (thousands more, same second, same bucket)

The counters restart at different values (9.7M, 13.1M, 16.6M…), so several pour routines are spinning on the same bucket at once. Each one produces about 100 warnings per 10 ms of waiting. On another server we saw attempts:12300000 within 718 ms for a single event on crowdsecurity/http-dos-swithcing-ua.

Consequences:

  • The log fills up quickly. With log_max_size: 20, one such burst consumes a large share of a log file in seconds.
  • CPU is wasted: one core per stuck pour routine for the duration of the stall.
  • The warning carries no actionable information: thousands of copies of the same message for one delayed event.
What did you expect to happen?
  • Block instead of spinning. In LIVE mode, the pour loop could wait on bucket.In <- parsed, <-bucket.done and <-ctx.Done() in a single blocking select. TIMEMACHINE mode would need a different approach, since the loop re-checks expiry there.
  • Warn once per event, or rate-limit the warning. For example, log once when the send finally succeeds, with the total wait time, instead of once every 100,000 iterations.
How can we reproduce it (as minimally and precisely as possible)?
  1. Run with buckets_routines > 1 and a leaky scenario such as crowdsecurity/http-crawl-non_statics.
  2. Send a burst of events for the same source IP fast enough that the bucket's leak routine is still busy when the next event arrives. A crawler hitting many non-static URLs does it.
  3. Watch crowdsec.log: the stuck for … sending event to … warning repeats every 100,000 iterations per waiting routine.

Relevant code (v1.8.1): pkg/leakybucket/manager_run.go, PourItemToBucket, which has the non-blocking select with default: failed_sent += 1; continue and the attempts%100000 == 0 warning:
https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/leakybucket/manager_run.go#L80-L150

Anything else we need to know?

Related: #1519 (same loop, CPU angle, closed in 2022).

Crowdsec version

v1.8.1-909b5157 (Windows MSI)

OS version

Windows Server 2019 / 2022

Lingua principale
Go
Stelle
15.1k
Fork
726
Merge medio
2g 51m
PR unite (30g)
18

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di crowdsecurity/crowdsec

Tutte le issue di crowdsecurity/crowdsec

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.