Gradual CPU leak: 5% to 345% over 13 days with auditd-logs parser (v1.7.8)

Abierto
  • ¿Algo no está bien?
#4,477 14 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
go
Área
backend, security

Línea de trabajo

Start with the reported acquisition config in /etc/crowdsec/config.yaml and the crowdsecurity/auditd-logs parser chain; compare CPU and memory use over time with the reported low log volume. The report does not name a code file or test, so locate the parser and relevant goroutines before investigating. Done means identifying and fixing the cause so CPU use remains stable during a sustained run.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

kind/bug needs/triage os/linux

Describe the bug
CrowdSec v1.7.8 gradually accumulates CPU usage over days/weeks until it consumes 345%+ CPU. The process starts at ~5% after restart and slowly climbs without bound. After 13 days of uptime, systemd reported the process consumed 3 weeks 2 days 14 hours of CPU time.

The log volume being processed is minimal (~8.5k lines from auditd, ~47 from auth.log), so this is not caused by high throughput.

To Reproduce

  1. Install CrowdSec v1.7.8 on Ubuntu 24.04
  2. Configure acquisition for /var/log/audit/audit.log (auditd) and /var/log/syslog
  3. Let it run for 7-14 days
  4. Observe CPU usage climbing from ~5% to 100%+ over time

Expected behavior
CPU usage should remain stable at ~5-10% regardless of uptime duration.

Evidence

Systemd stop output after 13 days:

May 20 01:49:00 Media-Server systemd[1]: crowdsec.service: Consumed 3w 2d 14h 58min 23.887s CPU time, 573.9M memory peak, 66.0M memory swap peak.

Process state at discovery:

root  3363534  345%  1.0  3110892 159888 ?  Rsl  May13 34007:09 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml

After restart:

root  2781659  5.8%  1.2  2385428 205484 ?  Ssl  01:49  0:32 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml

Acquisition metrics (showing minimal volume):

| Source                         | Lines read | Lines parsed | Lines unparsed |
| file:/var/log/audit/audit.log  | 8.59k      | 8.46k        | 128            |
| file:/var/log/auth.log         | 47         | 17           | 30             |

Acquisition config

filenames:
  - /var/log/audit/*.log
labels:
  type: auditd
source: file
---
filenames:
  - /var/log/messages
  - /var/log/syslog
  - /var/log/kern.log
labels:
  type: syslog
source: file
---
filenames:
  - /var/log/nginx/*.log
labels:
  type: nginx
source: file

Parsers installed

  • crowdsecurity/auditd-logs
  • crowdsecurity/sshd-logs
  • crowdsecurity/nginx-logs
  • crowdsecurity/http-logs

Environment

  • CrowdSec version: v1.7.8-debian-pragmatic-amd64-63227459
  • Go version: 1.26.2
  • OS: Ubuntu 24.04 LTS (kernel 6.8.0-111-generic)
  • Platform: linux/amd64 (Hetzner CPX42)
  • RAM: 16GB
  • Installed via: apt (debian package)

Workaround
Weekly or watchdog-based systemctl restart crowdsec prevents the leak from becoming critical. A watchdog script checking CPU > 50% for 30 minutes and restarting is our current mitigation.

Additional context

  • The leak appears to be in the Go runtime or parser goroutines, not in log processing volume
  • Memory peaks at ~574MB which is high for processing <10k lines
  • The auditd-logs parser chain (child-child-child-crowdsecurity/auditd-logs) may be involved
  • No panics or errors in journalctl logs — it just silently consumes more CPU over time
Lenguaje dominante
Go
Estrellas
15.1k
Forks
726
Merge medio
2 d 51 min
PR fusionados (30 d)
18

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de crowdsecurity/crowdsec

Todos los issues de crowdsecurity/crowdsec

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.