AWS: adds fail while object-storage writes are slow (integration's gap lock on Seq)

Abierto
#1,256 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@grempe ya está trabajando en esto.

Desde el 23/10/2026.

  • #1257 de @grempe — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
aws, go, mysql

Línea de trabajo

Start with storage/aws/aws.go at the referenced transaction boundaries and storage/aws/aws_test.go's MySQL sequencer tests. Reproduce the timeout with the supplied MySQL test command, then examine how transaction isolation and binary log settings affect the proposed change. Done means the regression test passes and the AWS driver works under STATEMENT, MIXED, and ROW binlog formats.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Reviewed and filed by me, a human. The investigation, repro and fix were developed with the assistance of an AI coding agent.

Summary: with the AWS driver, Add fails while writes to object storage are slow, although sequencing uses only MySQL.

  • consumeEntries reads the batches it integrates with SELECT … FROM Seq … FOR UPDATE, and keeps that transaction open while its consumeFunc writes the new tiles and bundles.
  • At MySQL's default isolation level, REPEATABLE READ, that read also locks the gap after the last batch it reads. So assignEntries' INSERT INTO Seq for the next batch waits for the integration to commit.
  • When the writes take longer than defaultAssignEntriesTimeout (2 s), the insert times out, and every entry in the batch fails with insert into seq: context deadline exceeded. The waiting insert holds SeqCoord's row lock, so the batches behind it fail too, with failed to read seqcoord: context deadline exceeded.

Where (main at a6e61ba):

#1207 describes the same lock spans from another side: a new instance's initDB waits on coordination rows that a running instance holds across object-storage and witness I/O. This one needs only one instance, and comes from the gap lock on Seq.

Repro. This test needs MySQL, like the driver's existing MySQL tests. It holds consumeEntries' consumeFunc open, as a slow write would, and sequences a batch meanwhile with the appender's 2 s. In a checkout of main:

git fetch https://github.com/grempe/tessera mysql-integrate-read-committed && git checkout FETCH_HEAD -- storage/aws/aws_test.go
go test -count=1 -run '^TestMySQLSequencerAssignWhileConsuming$' -v ./storage/aws/ -is_mysql_test_optional=false -mysql_uri='root:root@tcp(localhost:3306)/test_tessera'

On main, against MySQL 8.4 (=== RUN line omitted):

2026/10/23 15:33:51 ERROR failed to rollback Tx error="sql: transaction has already been committed or rolled back"
    aws_test.go:322: assignEntries while consuming: insert into seq: context deadline exceeded
--- FAIL: TestMySQLSequencerAssignWhileConsuming (2.08s)
FAIL
FAIL	github.com/transparency-dev/tessera/storage/aws	2.285s
FAIL

Seen end to end. We run the AWS driver against RustFS and MySQL 8.4, on main at 012e9e0, whose storage/aws matches a6e61ba.

  • With every tile and bundle write delayed by 3 s through a proxy, 100 of 200 adds (ten batches of 20) failed, in each of four runs.
  • In fault-injection tests at 100 adds/s, freezing the object store for 30 s, starving it of CPU or dropping 30% of its packets for 60 s, or cutting it off for 30 s each failed between 477 and 648 adds.

Fix: I'll open a PR from the same branch.

  • It begins consumeEntries' transaction at READ COMMITTED, where SELECT … FOR UPDATE locks only the rows it returns. The lock on IntCoord's row still keeps integrations one at a time, the batches read stay locked until commit, and consumeEntries already checks that they're contiguous. The driver's other transactions lock single rows by primary key, which takes no gap lock.
  • MySQL refuses writes at READ COMMITTED when it logs statements to its binary log (error 1665, binlog_format=STATEMENT). So newMySQLSequencer reads @@log_bin and @@binlog_format, and keeps the default isolation level, with a warning, where the log is statement-based or the settings can't be read. I checked that integration works under each of STATEMENT, MIXED and ROW (MySQL's default).

Would you rather have this as an aws.Config option than the automatic check? I haven't tried it on MySQL-compatible services such as Aurora or Vitess.

Lenguaje dominante
Go
Estrellas
262
Forks
64
Merge medio
1 d 6 h
PR fusionados (30 d)
20

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de transparency-dev/tessera

Todos los issues de transparency-dev/tessera

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.