Land and landsignal DLQs fail a `landing` batch that Runway goes on to merge

Open
  • Is something wrong?
#819 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
distributed-systems

Research direction

Start with submitqueue/orchestrator/controller/dlq/dlq.go and the interleaving described in the issue; then read spec/submitqueue/landoutcome/LandOutcome.tla and compare the design options. Add tests that interleave a DLQ with a late SUCCEEDED and confirm they fail before the fix. Done means the chosen design updates the model and meets all four acceptance criteria without allowing an incorrect terminal outcome.

Written by the indexing model from the issue text.

Description

bug

Problem

A batch is reported failed, and its requests error, even though Runway merged the change onto the target branch.

Cause

failBatch (submitqueue/orchestrator/controller/dlq/dlq.go) moves any non-terminal batch to failed, landing included. A landing batch has already handed its outcome to Runway, so the reconciler cannot know whether the merge happened. Two DLQs reach failBatch while the batch is landing:

  • submitqueue-land DLQ. The Runway publish is land's last step, so a lost ack, or an ambiguous publish error after the row was written, leaves Runway holding the MergeRequest. If later attempts then fail on a storage error, the message dead-letters.
  • landsignal DLQ. landsignal holds Runway's SUCCEEDED, but the batch compare-and-swap keeps failing on a storage fault. The DLQ payload contains the verdict, and the reconciler discards it.

When Runway's answer arrives later, landsignal sees a terminal batch, re-fans out, and acks. Nothing corrects the outcome.

Interleaving (TLC's shortest counterexample)

speculate:   batch speculating → landing, publish to submitqueue-land
land:        publish MergeRequest to runway-merge, then the attempt fails; retries exhaust → land DLQ
runway:      merges, publishes SUCCEEDED
land DLQ:    failBatch: landing → failed, fan out requests → error
landsignal:  batch already terminal → ack, outcome unchanged

Why the obvious fixes are not enough

Found by model checking spec/submitqueue/landoutcome/LandOutcome.tla, which checks 24 design combinations:

  • Skip landing in the DLQ and wait for Runway. This trades the wrong answer for a batch that never finishes, when Runway's answer is itself the message that dead-lettered.
  • Re-send the MergeRequest from the DLQ. This is unsound against today's Runway, for two reasons:
    • A stale FAILED can win. Runway's first attempt dead-letters without pushing, so its DLQ honestly answers FAILED. The re-send then merges on a second attempt, and the earlier FAILED is applied.
    • The replay is swallowed. A replayed answer reuses the request ID as its message ID and is deduped by the queue (the #352 class).
  • The only design that passes every property has four parts: the DLQ re-sends; Runway's DLQ checks the branch; Runway records its verdict per request ID; and answers do not dedup on the request ID alone. That gives Runway state, so cheaper designs are worth modelling first. Two candidates: the landsignal DLQ applies the verdict it holds, and land records "dispatched" before it publishes.

Acceptance criteria

  • A dead letter on submitqueue-land or landsignal never moves a landing batch to failed while the change can still land or has landed.
  • Every landing batch still reaches a terminal state.
  • Unit tests that interleave the DLQ with a late SUCCEEDED, confirmed to fail before the fix.
  • LandOutcome.tla updated to the chosen design.

Related: #818 (the same symptom through the speculate path).

Dominant language
Go
Stars
247
Forks
16
Avg merge
2d 1h
Merged PRs (30d)
125

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from uber/submitqueue

All issues in uber/submitqueue

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.