AWS: MySQL pools are never closed, so reopening a log in one process leaks goroutines and connections

Aperta
#1,251 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@grempe ci sta già lavorando.

Dal 9/11/2026.

  • #1276 di @grempe — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
aws, go, mysql
Ambito
backend, databases

Direzione di ricerca

Start with the sql.Open calls at storage/aws/aws.go:898 and storage/aws/antispam/aws.go:110, then compare how the GCP driver accepts caller-owned clients. The issue proposes caller-owned *sql.DB pools in aws.Config and AntispamOpts; review those APIs and their tests to see what needs updating. Done means repeated appender shutdown and reopen no longer accumulate MySQL connections and goroutines.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Reviewed and filed by me, a human. The investigation, repro and fix were developed with the assistance of an AI coding agent.

Summary: the AWS driver opens two MySQL pools, the sequencer's and the persistent antispam's, and never closes them.

  • Where: they're opened with sql.Open at aws.go:898 and antispam/aws.go:110. Neither the appender's shutdown function nor cancelling its context releases them.
  • In production, a process usually opens its log once, so this costs one goroutine per pool.
  • When a process opens a log again, as tests, soak tests and anything else that reopens a log in-process do, every reopen leaks each pool's database/sql connection-opener goroutine, plus every idle connection the pool was keeping, each with its driver watcher goroutine.

Measured on main at 012e9e0, against an S3-compatible store (RustFS) and MySQL 8.4:

  1. Open the AWS driver and an appender, then add one entry and await it.
  2. Call the appender's shutdown function, then cancel its context.
  3. Repeat 5 times in one process.
  4. Count goroutines and MySQL's Threads_connected 100 seconds after the last round, by which time the S3 client's idle HTTP connections have closed.
MaxIdleConns Pools Goroutines left MySQL connections left
unset (no idle connections; #1243) sequencer +5 0
unset sequencer + antispam +10 0
unset, with #1244 (database/sql's default of 2) sequencer + antispam +30 +20
64 sequencer +10 +5
64 sequencer + antispam +36 +26

At the last rate, MySQL's default max_connections of 151 runs out after about 30 reopens. Once #1244, or any non-zero MaxIdleConns, keeps connections idle, each leaked pool holds them open.

Repro. It needs a MySQL server, where it creates two databases, and an S3 or S3-compatible bucket. AWS credentials come from the usual environment variables. In a checkout of main:

git fetch https://github.com/grempe/tessera repro-mysql-pool-leak && git checkout FETCH_HEAD -- repro/reopen/main.go
go run ./repro/reopen -mysql_dsn='root:root@tcp(localhost:3306)/' -bucket=<bucket> -s3_endpoint=<endpoint, or empty for AWS> -idle=64 -antispam

Run that way against RustFS and MySQL 8.4 (time= log lines omitted). The per-round counts, which include the S3 client's idle HTTP connections, vary a little between runs; the counts after the wait were the same in each run:

MaxIdleConns=64 antispam=true
  after round  1: +11 goroutines, +5 MySQL connections
  after round  2: +21 goroutines, +11 MySQL connections
  after round  3: +32 goroutines, +16 MySQL connections
  after round  4: +41 goroutines, +21 MySQL connections
  after round  5: +52 goroutines, +26 MySQL connections
  1m40s after the last round: +36 goroutines, +26 MySQL connections
  goroutines, by function:
     27 github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1
     11 database/sql.(*DB).connectionOpener

Possible directions:

  1. Accept a caller-owned *sql.DB in aws.Config and AntispamOpts. This mirrors the GCP driver: its Config takes a GCSClient and a SpannerClient, and since #1054 its AntispamOpts takes a SpannerClient that Tessera never closes. The caller then decides the pool's lifetime, and gets control of its settings, and of sharing it.
  2. Close Tessera's own pools once the driver's background work has stopped. That needs the driver to know when its goroutines have finished, which is what #1111 asks for.
  3. A Close method on the driver, called after the appender's shutdown function.

I'll open a PR for option 1. With caller-owned pools passed in, the same 5 reopens with MaxIdleConns 64 and the antispam leave +6 goroutines and +6 MySQL connections in total: the two pools' own idle connections, reused by every round rather than growing. Options 2 and 3 would still be needed for pools that Tessera opens itself.

Lingua principale
Go
Stelle
262
Fork
64
Merge medio
1g 6h
PR unite (30g)
20

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di transparency-dev/tessera

Tutte le issue di transparency-dev/tessera

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.