AWS: adds fail while object-storage writes are slow (integration's gap lock on Seq)
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 35/100
Línea de trabajo
Start with storage/aws/aws.go at the referenced transaction boundaries and storage/aws/aws_test.go's MySQL sequencer tests. Reproduce the timeout with the supplied MySQL test command, then examine how transaction isolation and binary log settings affect the proposed change. Done means the regression test passes and the AWS driver works under STATEMENT, MIXED, and ROW binlog formats.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Reviewed and filed by me, a human. The investigation, repro and fix were developed with the assistance of an AI coding agent.
Summary: with the AWS driver, Add fails while writes to object storage are slow, although sequencing uses only MySQL.
consumeEntriesreads the batches it integrates withSELECT … FROM Seq … FOR UPDATE, and keeps that transaction open while itsconsumeFuncwrites the new tiles and bundles.- At MySQL's default isolation level, REPEATABLE READ, that read also locks the gap after the last batch it reads. So
assignEntries'INSERT INTO Seqfor the next batch waits for the integration to commit. - When the writes take longer than
defaultAssignEntriesTimeout(2 s), the insert times out, and every entry in the batch fails withinsert into seq: context deadline exceeded. The waiting insert holdsSeqCoord's row lock, so the batches behind it fail too, withfailed to read seqcoord: context deadline exceeded.
Where (main at a6e61ba):
consumeEntriesbegins its transaction with the default options (aws.go:1161), readsSeqFOR UPDATE(aws.go:1185), and callsf(aws.go:1229) before it commits (aws.go:1248).assignEntrieslocksSeqCoord's row (aws.go:1105) and then inserts the batch (aws.go:1136), withindefaultAssignEntriesTimeout(aws.go:92, applied at aws.go:250).
#1207 describes the same lock spans from another side: a new instance's initDB waits on coordination rows that a running instance holds across object-storage and witness I/O. This one needs only one instance, and comes from the gap lock on Seq.
Repro. This test needs MySQL, like the driver's existing MySQL tests. It holds consumeEntries' consumeFunc open, as a slow write would, and sequences a batch meanwhile with the appender's 2 s. In a checkout of main:
git fetch https://github.com/grempe/tessera mysql-integrate-read-committed && git checkout FETCH_HEAD -- storage/aws/aws_test.go
go test -count=1 -run '^TestMySQLSequencerAssignWhileConsuming$' -v ./storage/aws/ -is_mysql_test_optional=false -mysql_uri='root:root@tcp(localhost:3306)/test_tessera'
On main, against MySQL 8.4 (=== RUN line omitted):
2026/10/23 15:33:51 ERROR failed to rollback Tx error="sql: transaction has already been committed or rolled back"
aws_test.go:322: assignEntries while consuming: insert into seq: context deadline exceeded
--- FAIL: TestMySQLSequencerAssignWhileConsuming (2.08s)
FAIL
FAIL github.com/transparency-dev/tessera/storage/aws 2.285s
FAIL
Seen end to end. We run the AWS driver against RustFS and MySQL 8.4, on main at 012e9e0, whose storage/aws matches a6e61ba.
- With every tile and bundle write delayed by 3 s through a proxy, 100 of 200 adds (ten batches of 20) failed, in each of four runs.
- In fault-injection tests at 100 adds/s, freezing the object store for 30 s, starving it of CPU or dropping 30% of its packets for 60 s, or cutting it off for 30 s each failed between 477 and 648 adds.
Fix: I'll open a PR from the same branch.
- It begins
consumeEntries' transaction at READ COMMITTED, whereSELECT … FOR UPDATElocks only the rows it returns. The lock on IntCoord's row still keeps integrations one at a time, the batches read stay locked until commit, andconsumeEntriesalready checks that they're contiguous. The driver's other transactions lock single rows by primary key, which takes no gap lock. - MySQL refuses writes at READ COMMITTED when it logs statements to its binary log (error 1665,
binlog_format=STATEMENT). SonewMySQLSequencerreads@@log_binand@@binlog_format, and keeps the default isolation level, with a warning, where the log is statement-based or the settings can't be read. I checked that integration works under each of STATEMENT, MIXED and ROW (MySQL's default).
Would you rather have this as an aws.Config option than the automatic check? I haven't tried it on MySQL-compatible services such as Aurora or Vitess.
- Lenguaje dominante
- Go
- Estrellas
- 262
- Forks
- 64
- Merge medio
- 1 d 6 h
- PR fusionados (30 d)
- 20
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de transparency-dev/tessera
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
transparency-dev/tessera#1286 ·
Los mantenedores suelen responder en 1 día
-
TestPosixMirrorIntegration fails on macOS: its burst of connections overflows the listen queuePosiblemente ocupada @grempe la tomó hace 30 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
transparency-dev/tessera#1253 ·
Los mantenedores suelen responder en 1 día
-
Fix tlog-mirror log origin length prefixPosiblemente ocupada @phbnf la tomó hace 32 días. Abiertoenhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
transparency-dev/tessera#1233 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
AWS: MySQL pools are never closed, so reopening a log in one process leaks goroutines and connectionsPosiblemente ocupada @grempe la tomó hace 13 días. Abierto
Dificultad 4/5 1-2 días Aptitud para principiantes 48/100
transparency-dev/tessera#1251 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
PublicationAwaiter reads the checkpoint every poll period even when nothing is waitingPosiblemente ocupada @grempe la tomó hace 31 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 35/100
transparency-dev/tessera#1249 ·
Los mantenedores suelen responder en 1 día
Todos los issues de transparency-dev/tessera
Issues similares
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 2 días
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
seaweedfs/seaweed-block#443 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
version-fox/vfox#825 ·
Los mantenedores suelen responder en 1 día
-
HTTP `batch.Send` returns nil and inserts no rows when the block does not encodePosiblemente ocupada @Indian-boult la tomó hoy. Abiertobug queries
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
ClickHouse/clickhouse-go#2155 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día