AWS: adds fail while object-storage writes are slow (integration's gap lock on Seq)

Aperta
#1,256 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@grempe ci sta già lavorando.

Dal 23/10/2026.

  • #1257 di @grempe — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
aws, go, mysql
Ambito
backend, databases

Direzione di ricerca

Start with storage/aws/aws.go at the referenced transaction boundaries and storage/aws/aws_test.go's MySQL sequencer tests. Reproduce the timeout with the supplied MySQL test command, then examine how transaction isolation and binary log settings affect the proposed change. Done means the regression test passes and the AWS driver works under STATEMENT, MIXED, and ROW binlog formats.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Reviewed and filed by me, a human. The investigation, repro and fix were developed with the assistance of an AI coding agent.

Summary: with the AWS driver, Add fails while writes to object storage are slow, although sequencing uses only MySQL.

  • consumeEntries reads the batches it integrates with SELECT … FROM Seq … FOR UPDATE, and keeps that transaction open while its consumeFunc writes the new tiles and bundles.
  • At MySQL's default isolation level, REPEATABLE READ, that read also locks the gap after the last batch it reads. So assignEntries' INSERT INTO Seq for the next batch waits for the integration to commit.
  • When the writes take longer than defaultAssignEntriesTimeout (2 s), the insert times out, and every entry in the batch fails with insert into seq: context deadline exceeded. The waiting insert holds SeqCoord's row lock, so the batches behind it fail too, with failed to read seqcoord: context deadline exceeded.

Where (main at a6e61ba):

#1207 describes the same lock spans from another side: a new instance's initDB waits on coordination rows that a running instance holds across object-storage and witness I/O. This one needs only one instance, and comes from the gap lock on Seq.

Repro. This test needs MySQL, like the driver's existing MySQL tests. It holds consumeEntries' consumeFunc open, as a slow write would, and sequences a batch meanwhile with the appender's 2 s. In a checkout of main:

git fetch https://github.com/grempe/tessera mysql-integrate-read-committed && git checkout FETCH_HEAD -- storage/aws/aws_test.go
go test -count=1 -run '^TestMySQLSequencerAssignWhileConsuming$' -v ./storage/aws/ -is_mysql_test_optional=false -mysql_uri='root:root@tcp(localhost:3306)/test_tessera'

On main, against MySQL 8.4 (=== RUN line omitted):

2026/10/23 15:33:51 ERROR failed to rollback Tx error="sql: transaction has already been committed or rolled back"
    aws_test.go:322: assignEntries while consuming: insert into seq: context deadline exceeded
--- FAIL: TestMySQLSequencerAssignWhileConsuming (2.08s)
FAIL
FAIL	github.com/transparency-dev/tessera/storage/aws	2.285s
FAIL

Seen end to end. We run the AWS driver against RustFS and MySQL 8.4, on main at 012e9e0, whose storage/aws matches a6e61ba.

  • With every tile and bundle write delayed by 3 s through a proxy, 100 of 200 adds (ten batches of 20) failed, in each of four runs.
  • In fault-injection tests at 100 adds/s, freezing the object store for 30 s, starving it of CPU or dropping 30% of its packets for 60 s, or cutting it off for 30 s each failed between 477 and 648 adds.

Fix: I'll open a PR from the same branch.

  • It begins consumeEntries' transaction at READ COMMITTED, where SELECT … FOR UPDATE locks only the rows it returns. The lock on IntCoord's row still keeps integrations one at a time, the batches read stay locked until commit, and consumeEntries already checks that they're contiguous. The driver's other transactions lock single rows by primary key, which takes no gap lock.
  • MySQL refuses writes at READ COMMITTED when it logs statements to its binary log (error 1665, binlog_format=STATEMENT). So newMySQLSequencer reads @@log_bin and @@binlog_format, and keeps the default isolation level, with a warning, where the log is statement-based or the settings can't be read. I checked that integration works under each of STATEMENT, MIXED and ROW (MySQL's default).

Would you rather have this as an aws.Config option than the automatic check? I haven't tried it on MySQL-compatible services such as Aurora or Vitess.

Lingua principale
Go
Stelle
262
Fork
64
Merge medio
1g 6h
PR unite (30g)
20

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di transparency-dev/tessera

Tutte le issue di transparency-dev/tessera

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.