AppSec: SIGHUP reload deadlocks in shutdown — handlers block on InChan after runners exit, server.Shutdown waits forever

Abierto
  • ¿Algo no está bien?
#4,739 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
72/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go
Área
backend, security

Línea de trabajo

Start at pkg/acquisition/modules/appsec/config.go appsecHandler around line 468, where the unconditional send w.InChan <- parsedRequest blocks, then read pkg/acquisition/modules/appsec/run.go listenAndServe (the unbounded server.Shutdown) and cmd/crowdsec/serve.go ShutdownCrowdsecRoutines/shutdownCrowdsec to see the shutdown order. The issue lists three candidate fixes; pick one and check existing tests under pkg/acquisition/modules/appsec/ for shutdown/HTTP coverage to extend. Done means a SIGHUP with at least one in-flight AppSec request completes the reload instead of hanging, verified by a test or a reproduced reload run.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

needs/kind needs/triage
What happened?

A SIGHUP reload (sent by the packaged crowdsec-hubupdate after cscli hub upgrade found updates) never completed. The agent stopped all acquisition — file datasources and AppSec — and never started again. The process stayed alive, systemd kept reporting active (running), and nothing restarted it. WAF and log-based detection were both down for 8 hours until a manual systemctl restart (which itself had to fall back to SIGKILL after TimeoutStopSec).

The last lines in crowdsec.log, after which nothing is logged at all:

level=warning msg="SIGHUP received, reloading"
level=info msg="File datasource stopping" ...
level=info msg="Appsec Runner is dying" ...

There are no bind errors and no Error shutting down Appsec server — the reload never reaches the point of starting the new acquisition.

Goroutine dump of the hung process

Taken from /debug/pprof/goroutine?debug=1 before restarting. Relevant stacks (line numbers match commit 909b5157, the source of the 1.8.1 Debian package):

9 @ ...
#  appsec.(*Source).appsecHandler+0x6f6     pkg/acquisition/modules/appsec/config.go:468
#  net/http.HandlerFunc.ServeHTTP            net/http/server.go:2286
#  net/http.(*conn).serve                    net/http/server.go:2073

1 @ ...
#  appsec.(*Source).listenAndServe+0x430     pkg/acquisition/modules/appsec/run.go:104
#  appsec.(*Source).StreamingAcquisition.func1  pkg/acquisition/modules/appsec/run.go:153

1 @ ...
#  main.ShutdownCrowdsecRoutines+0xc8        cmd/crowdsec/serve.go:113
#  main.serveCrowdsec.func1                  cmd/crowdsec/crowdsec.go:243

1 @ ...
#  main.shutdownCrowdsec+0x53                cmd/crowdsec/serve.go:180
#  main.shutdown                             cmd/crowdsec/serve.go:191
#  main.HandleSignals.func1                  cmd/crowdsec/serve.go:266

No AppSec runner goroutines are left.

Analysis

The shutdown order lets in-flight AppSec requests block forever:

  1. acquisTomb.Kill(nil) — every goroutine of the AppSec source sees Dying() at the same time.
  2. The runners return immediately (appsec_runner.go:564), so nothing reads inChan any more.
  3. Requests that were already accepted by the HTTP server block in appsecHandler on the unconditional send w.InChan <- parsedRequest (config.go:468). There is no select with a cancellation case, so they can never return.
  4. listenAndServe calls w.server.Shutdown(ctx) (run.go:104). Shutdown waits for active connections to finish, and ctx has no deadline, so it waits forever.
  5. acquisTomb.Wait() (serve.go:113) and crowdsecTomb.Wait() (serve.go:180) never return; the reload never proceeds.

It only takes one request to be in flight at the moment the runners exit. On a bouncer with steady traffic (~7–8 inspected requests/s here) that is close to certain, so this is not a rare race in practice — it just only shows up on the nights when the hub actually upgrades something.

This is related to #4545 but is a different failure: there the shutdown completes and the listener leaks / fails to re-bind afterwards, here the shutdown itself never completes and there are no bind errors.

What did you expect to happen?

The reload completes, or — if AppSec cannot be stopped cleanly — the process fails loudly so a supervisor can restart it, instead of staying alive with every datasource stopped.

Possible fixes

Any one of these would break the cycle:

  • In appsecHandler, make the send to InChan (and the receive from ResponseChannel right after it) a select that also returns on shutdown, answering the bouncer with an error.
  • Call server.Shutdown with a bounded context and fall back to server.Close() on timeout.
  • Shut the HTTP server down and let the runners drain inChan before stopping the runners.
How can we reproduce it?

Enable AppSec, keep a steady stream of requests going through a bouncer, and send SIGHUP (or systemctl reload crowdsec). The hang needs at least one request to be inside appsecHandler when the runners exit.

Workaround

We replaced reload with restart both in a drop-in for crowdsec-hubupdate.service and in our configuration management handlers: a hung stop is then killed by systemd after TimeoutStopSec.

Crowdsec version
version: v1.8.1-debian-pragmatic-amd64-909b5157
Codename: alphaga
BuildDate: 2026-09-03_10:51:51

Installed from the crowdsec Debian package (1.8.1). AppSec enabled with the bot-challenge collections, OpenResty bouncer in front.

OS version

Ubuntu 22.04.5 LTS

Enabled collections and parsers

crowdsecurity/linux, sshd, nginx, base-http-scenarios, http-cve, http-dos, appsec-virtual-patching, appsec-generic-rules, appsec-bot-challenge-permissive, appsec-bot-challenge-good-bots, appsec-bot-challenge-exclude-paths, appsec-bot-challenge-scoring, plus a few local scenarios and parsers.

Acquisition config

File datasources on nginx and GitLab logs, plus:

source: appsec
listen_addr: <private address>:7422
appsec_configs:
  - crowdsecurity/appsec-default
  - crowdsecurity/appsec-bot-challenge-scoring
  - crowdsecurity/appsec-bot-challenge-scoring-permissive
  # ... bot-challenge exclusions ...
Lenguaje dominante
Go
Estrellas
15.1k
Forks
726
Merge medio
1 d 10 h
PR fusionados (30 d)
15

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de crowdsecurity/crowdsec

Todos los issues de crowdsecurity/crowdsec

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.