AWS: adds fail while object-storage writes are slow (integration's gap lock on Seq)
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 35/100
Direzione di ricerca
Start with storage/aws/aws.go at the referenced transaction boundaries and storage/aws/aws_test.go's MySQL sequencer tests. Reproduce the timeout with the supplied MySQL test command, then examine how transaction isolation and binary log settings affect the proposed change. Done means the regression test passes and the AWS driver works under STATEMENT, MIXED, and ROW binlog formats.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Reviewed and filed by me, a human. The investigation, repro and fix were developed with the assistance of an AI coding agent.
Summary: with the AWS driver, Add fails while writes to object storage are slow, although sequencing uses only MySQL.
consumeEntriesreads the batches it integrates withSELECT … FROM Seq … FOR UPDATE, and keeps that transaction open while itsconsumeFuncwrites the new tiles and bundles.- At MySQL's default isolation level, REPEATABLE READ, that read also locks the gap after the last batch it reads. So
assignEntries'INSERT INTO Seqfor the next batch waits for the integration to commit. - When the writes take longer than
defaultAssignEntriesTimeout(2 s), the insert times out, and every entry in the batch fails withinsert into seq: context deadline exceeded. The waiting insert holdsSeqCoord's row lock, so the batches behind it fail too, withfailed to read seqcoord: context deadline exceeded.
Where (main at a6e61ba):
consumeEntriesbegins its transaction with the default options (aws.go:1161), readsSeqFOR UPDATE(aws.go:1185), and callsf(aws.go:1229) before it commits (aws.go:1248).assignEntrieslocksSeqCoord's row (aws.go:1105) and then inserts the batch (aws.go:1136), withindefaultAssignEntriesTimeout(aws.go:92, applied at aws.go:250).
#1207 describes the same lock spans from another side: a new instance's initDB waits on coordination rows that a running instance holds across object-storage and witness I/O. This one needs only one instance, and comes from the gap lock on Seq.
Repro. This test needs MySQL, like the driver's existing MySQL tests. It holds consumeEntries' consumeFunc open, as a slow write would, and sequences a batch meanwhile with the appender's 2 s. In a checkout of main:
git fetch https://github.com/grempe/tessera mysql-integrate-read-committed && git checkout FETCH_HEAD -- storage/aws/aws_test.go
go test -count=1 -run '^TestMySQLSequencerAssignWhileConsuming$' -v ./storage/aws/ -is_mysql_test_optional=false -mysql_uri='root:root@tcp(localhost:3306)/test_tessera'
On main, against MySQL 8.4 (=== RUN line omitted):
2026/10/23 15:33:51 ERROR failed to rollback Tx error="sql: transaction has already been committed or rolled back"
aws_test.go:322: assignEntries while consuming: insert into seq: context deadline exceeded
--- FAIL: TestMySQLSequencerAssignWhileConsuming (2.08s)
FAIL
FAIL github.com/transparency-dev/tessera/storage/aws 2.285s
FAIL
Seen end to end. We run the AWS driver against RustFS and MySQL 8.4, on main at 012e9e0, whose storage/aws matches a6e61ba.
- With every tile and bundle write delayed by 3 s through a proxy, 100 of 200 adds (ten batches of 20) failed, in each of four runs.
- In fault-injection tests at 100 adds/s, freezing the object store for 30 s, starving it of CPU or dropping 30% of its packets for 60 s, or cutting it off for 30 s each failed between 477 and 648 adds.
Fix: I'll open a PR from the same branch.
- It begins
consumeEntries' transaction at READ COMMITTED, whereSELECT … FOR UPDATElocks only the rows it returns. The lock on IntCoord's row still keeps integrations one at a time, the batches read stay locked until commit, andconsumeEntriesalready checks that they're contiguous. The driver's other transactions lock single rows by primary key, which takes no gap lock. - MySQL refuses writes at READ COMMITTED when it logs statements to its binary log (error 1665,
binlog_format=STATEMENT). SonewMySQLSequencerreads@@log_binand@@binlog_format, and keeps the default isolation level, with a warning, where the log is statement-based or the settings can't be read. I checked that integration works under each of STATEMENT, MIXED and ROW (MySQL's default).
Would you rather have this as an aws.Config option than the automatic check? I haven't tried it on MySQL-compatible services such as Aurora or Vitess.
- Lingua principale
- Go
- Stelle
- 262
- Fork
- 64
- Merge medio
- 1g 6h
- PR unite (30g)
- 20
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di transparency-dev/tessera
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
transparency-dev/tessera#1286 ·
I maintainer di solito rispondono entro 1 giorno
-
TestPosixMirrorIntegration fails on macOS: its burst of connections overflows the listen queueForse già presa @grempe l’ha presa 30 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
transparency-dev/tessera#1253 ·
I maintainer di solito rispondono entro 1 giorno
-
Fix tlog-mirror log origin length prefixForse già presa @phbnf l’ha presa 32 giorni fa. Apertaenhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
transparency-dev/tessera#1233 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
AWS: MySQL pools are never closed, so reopening a log in one process leaks goroutines and connectionsForse già presa @grempe l’ha presa 13 giorni fa. Aperta
Difficoltà 4/5 1-2 giorni Idoneità per principianti 48/100
transparency-dev/tessera#1251 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
PublicationAwaiter reads the checkpoint every poll period even when nothing is waitingForse già presa @grempe l’ha presa 31 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 35/100
transparency-dev/tessera#1249 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di transparency-dev/tessera
Issue simili
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 2 giorni
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
seaweedfs/seaweed-block#443 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
version-fox/vfox#825 ·
I maintainer di solito rispondono entro 1 giorno
-
HTTP `batch.Send` returns nil and inserts no rows when the block does not encodeForse già presa @Indian-boult l’ha presa oggi. Apertabug queries
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
ClickHouse/clickhouse-go#2155 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno