AWS: MySQL pools are never closed, so reopening a log in one process leaks goroutines and connections
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 48/100
Direzione di ricerca
Start with the sql.Open calls at storage/aws/aws.go:898 and storage/aws/antispam/aws.go:110, then compare how the GCP driver accepts caller-owned clients. The issue proposes caller-owned *sql.DB pools in aws.Config and AntispamOpts; review those APIs and their tests to see what needs updating. Done means repeated appender shutdown and reopen no longer accumulate MySQL connections and goroutines.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Reviewed and filed by me, a human. The investigation, repro and fix were developed with the assistance of an AI coding agent.
Summary: the AWS driver opens two MySQL pools, the sequencer's and the persistent antispam's, and never closes them.
- Where: they're opened with
sql.Openat aws.go:898 and antispam/aws.go:110. Neither the appender's shutdown function nor cancelling its context releases them. - In production, a process usually opens its log once, so this costs one goroutine per pool.
- When a process opens a log again, as tests, soak tests and anything else that reopens a log in-process do, every reopen leaks each pool's
database/sqlconnection-opener goroutine, plus every idle connection the pool was keeping, each with its driver watcher goroutine.
Measured on main at 012e9e0, against an S3-compatible store (RustFS) and MySQL 8.4:
- Open the AWS driver and an appender, then add one entry and await it.
- Call the appender's shutdown function, then cancel its context.
- Repeat 5 times in one process.
- Count goroutines and MySQL's
Threads_connected100 seconds after the last round, by which time the S3 client's idle HTTP connections have closed.
MaxIdleConns |
Pools | Goroutines left | MySQL connections left |
|---|---|---|---|
| unset (no idle connections; #1243) | sequencer | +5 | 0 |
| unset | sequencer + antispam | +10 | 0 |
unset, with #1244 (database/sql's default of 2) |
sequencer + antispam | +30 | +20 |
| 64 | sequencer | +10 | +5 |
| 64 | sequencer + antispam | +36 | +26 |
At the last rate, MySQL's default max_connections of 151 runs out after about 30 reopens. Once #1244, or any non-zero MaxIdleConns, keeps connections idle, each leaked pool holds them open.
Repro. It needs a MySQL server, where it creates two databases, and an S3 or S3-compatible bucket. AWS credentials come from the usual environment variables. In a checkout of main:
git fetch https://github.com/grempe/tessera repro-mysql-pool-leak && git checkout FETCH_HEAD -- repro/reopen/main.go
go run ./repro/reopen -mysql_dsn='root:root@tcp(localhost:3306)/' -bucket=<bucket> -s3_endpoint=<endpoint, or empty for AWS> -idle=64 -antispam
Run that way against RustFS and MySQL 8.4 (time= log lines omitted). The per-round counts, which include the S3 client's idle HTTP connections, vary a little between runs; the counts after the wait were the same in each run:
MaxIdleConns=64 antispam=true
after round 1: +11 goroutines, +5 MySQL connections
after round 2: +21 goroutines, +11 MySQL connections
after round 3: +32 goroutines, +16 MySQL connections
after round 4: +41 goroutines, +21 MySQL connections
after round 5: +52 goroutines, +26 MySQL connections
1m40s after the last round: +36 goroutines, +26 MySQL connections
goroutines, by function:
27 github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1
11 database/sql.(*DB).connectionOpener
Possible directions:
- Accept a caller-owned
*sql.DBinaws.ConfigandAntispamOpts. This mirrors the GCP driver: itsConfigtakes aGCSClientand aSpannerClient, and since #1054 itsAntispamOptstakes aSpannerClientthat Tessera never closes. The caller then decides the pool's lifetime, and gets control of its settings, and of sharing it. - Close Tessera's own pools once the driver's background work has stopped. That needs the driver to know when its goroutines have finished, which is what #1111 asks for.
- A
Closemethod on the driver, called after the appender's shutdown function.
I'll open a PR for option 1. With caller-owned pools passed in, the same 5 reopens with MaxIdleConns 64 and the antispam leave +6 goroutines and +6 MySQL connections in total: the two pools' own idle connections, reused by every round rather than growing. Options 2 and 3 would still be needed for pools that Tessera opens itself.
- Lingua principale
- Go
- Stelle
- 262
- Fork
- 64
- Merge medio
- 1g 6h
- PR unite (30g)
- 20
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di transparency-dev/tessera
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
transparency-dev/tessera#1286 ·
I maintainer di solito rispondono entro 1 giorno
-
TestPosixMirrorIntegration fails on macOS: its burst of connections overflows the listen queueForse già presa @grempe l’ha presa 30 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
transparency-dev/tessera#1253 ·
I maintainer di solito rispondono entro 1 giorno
-
Fix tlog-mirror log origin length prefixForse già presa @phbnf l’ha presa 32 giorni fa. Apertaenhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
transparency-dev/tessera#1233 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
AWS: adds fail while object-storage writes are slow (integration's gap lock on Seq)Forse già presa @grempe l’ha presa 30 giorni fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
transparency-dev/tessera#1256 ·
I maintainer di solito rispondono entro 1 giorno
-
PublicationAwaiter reads the checkpoint every poll period even when nothing is waitingForse già presa @grempe l’ha presa 31 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 35/100
transparency-dev/tessera#1249 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di transparency-dev/tessera
Issue simili
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 2 giorni
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
seaweedfs/seaweed-block#443 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
version-fox/vfox#825 ·
I maintainer di solito rispondono entro 1 giorno
-
HTTP `batch.Send` returns nil and inserts no rows when the block does not encodeForse già presa @Indian-boult l’ha presa oggi. Apertabug queries
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
ClickHouse/clickhouse-go#2155 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno