Land and landsignal DLQs fail a `landing` batch that Runway goes on to merge

Offen
  • Stimmt etwas nicht?
#819 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
go
Bereich
distributed-systems

Rechercherichtung

Start with submitqueue/orchestrator/controller/dlq/dlq.go and the interleaving described in the issue; then read spec/submitqueue/landoutcome/LandOutcome.tla and compare the design options. Add tests that interleave a DLQ with a late SUCCEEDED and confirm they fail before the fix. Done means the chosen design updates the model and meets all four acceptance criteria without allowing an incorrect terminal outcome.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

bug

Problem

A batch is reported failed, and its requests error, even though Runway merged the change onto the target branch.

Cause

failBatch (submitqueue/orchestrator/controller/dlq/dlq.go) moves any non-terminal batch to failed, landing included. A landing batch has already handed its outcome to Runway, so the reconciler cannot know whether the merge happened. Two DLQs reach failBatch while the batch is landing:

  • submitqueue-land DLQ. The Runway publish is land's last step, so a lost ack, or an ambiguous publish error after the row was written, leaves Runway holding the MergeRequest. If later attempts then fail on a storage error, the message dead-letters.
  • landsignal DLQ. landsignal holds Runway's SUCCEEDED, but the batch compare-and-swap keeps failing on a storage fault. The DLQ payload contains the verdict, and the reconciler discards it.

When Runway's answer arrives later, landsignal sees a terminal batch, re-fans out, and acks. Nothing corrects the outcome.

Interleaving (TLC's shortest counterexample)

speculate:   batch speculating → landing, publish to submitqueue-land
land:        publish MergeRequest to runway-merge, then the attempt fails; retries exhaust → land DLQ
runway:      merges, publishes SUCCEEDED
land DLQ:    failBatch: landing → failed, fan out requests → error
landsignal:  batch already terminal → ack, outcome unchanged

Why the obvious fixes are not enough

Found by model checking spec/submitqueue/landoutcome/LandOutcome.tla, which checks 24 design combinations:

  • Skip landing in the DLQ and wait for Runway. This trades the wrong answer for a batch that never finishes, when Runway's answer is itself the message that dead-lettered.
  • Re-send the MergeRequest from the DLQ. This is unsound against today's Runway, for two reasons:
    • A stale FAILED can win. Runway's first attempt dead-letters without pushing, so its DLQ honestly answers FAILED. The re-send then merges on a second attempt, and the earlier FAILED is applied.
    • The replay is swallowed. A replayed answer reuses the request ID as its message ID and is deduped by the queue (the #352 class).
  • The only design that passes every property has four parts: the DLQ re-sends; Runway's DLQ checks the branch; Runway records its verdict per request ID; and answers do not dedup on the request ID alone. That gives Runway state, so cheaper designs are worth modelling first. Two candidates: the landsignal DLQ applies the verdict it holds, and land records "dispatched" before it publishes.

Acceptance criteria

  • A dead letter on submitqueue-land or landsignal never moves a landing batch to failed while the change can still land or has landed.
  • Every landing batch still reaches a terminal state.
  • Unit tests that interleave the DLQ with a late SUCCEEDED, confirmed to fail before the fix.
  • LandOutcome.tla updated to the chosen design.

Related: #818 (the same symptom through the speculate path).

Vorherrschende Sprache
Go
Sterne
247
Forks
16
Ø Merge
2 T. 1 Std.
Gemergte PRs (30 T.)
125

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus uber/submitqueue

Alle Issues in uber/submitqueue

Ähnliche Issues

Weitere Issues zu Go

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.