jobs: follow-ups to the River job deadline fix (#632): Close without a drain, sweep without a pass cap, slower sync rescue

Aperta
  • Qualcosa non va?
#636 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
backend, devops

Direzione di ricerca

Start with the referenced shutdown paths in internal/jobs/river.go and cmd/openstorage/main.go, then trace the sweep in internal/jobs/sweep.go and blob.WithRenewedCover in internal/blob/blob.go. The issue lists three separate follow-ups but no selected scope or acceptance criteria; clarify which behavior to address before changing code. Done should include a focused fix and coverage for the affected shutdown, sweep, or rescue behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

storage-gc tracking

The fix for #632 gives every River job a 2-hour deadline and rescues a crash-orphaned job after 2 h 30 min. Two limits stay. I found both by reading the code. I did not run either at runtime.

1. River.Close can wait up to the job deadline on serve's startup-failure path

  • River.Close calls client.Stop(context.Background()), which waits for every in-flight job (internal/jobs/river.go, Close).
  • Two kinds of exit reach Close without DrainSyncRuns first, while the run-on-start sweep or another job can be running:
    • serve's wiring fails after NewRiver started the client: cmd/openstorage/wire.go calls queueCleanup().
    • runServe returns early after defer cleanup() (cmd/openstorage/main.go): a setup error such as NewGitRateLimit, the HTTP listener stopping (serveErr), or the metrics listener stopping (metricsErr).
  • Before #632 the 1-minute deadline ended such a job early. Now the job can run until its own budget or the 2-hour deadline ends, plus the sweep's 15-minute detached recovery. So such an exit can take that long.
  • Signal shutdown is not affected: DrainSyncRuns cancels the remaining jobs when its context ends (cmd/openstorage/main.go, the drain after the termination signal).
  • Fix idea: in Close, stop with a bounded context, then StopAndCancel.

2. The orphan sweep has no pass cap and no resume cursor

  • In Postgres mode the 2-hour River deadline is now the only bound on one sweep pass (SweepSchedule.runOnce in internal/jobs/sweep.go runs the sweep and then the repo purge on the job context). Local mode has no bound.
  • A fleet whose sweep needs more than 2 hours never finishes a pass in Postgres mode, and the purge after it never runs. Each pass restarts from the beginning.
  • A sweep stuck in the cover-handoff retry loop of blob.WithRenewedCover (internal/blob/blob.go, the final loop, which retries while renewals succeed by design) can also outlive the 2 h 30 min rescue horizon. The next sweep then runs beside it, which the sweep tolerates (internal/jobs/sweep.go, "A concurrent sweep purged this entry first").
  • Trigger: the first fleet whose sweep takes more than about 1 hour, or a resumable sweep design.

3. Crash rescue for sync and push now waits 2 h 30 min instead of 1 hour

  • After a crash, an orphaned upstream_sync or upstream_push row stays running until River rescues it. It is unique including the running state (internal/jobs/river.go, syncArgs and pushArgs InsertOpts), so new requests for that repo deduplicate against it. They stay marked, so the rescued run catches them up, but only after 2 h 30 min.
  • River skips a job until both the client cutoff (RescueStuckJobsAfter) and the kind's own timeout have passed (vendor/github.com/riverqueue/river/internal/maintenance/job_rescuer.go, the timeout check in the retry decision). A short client cutoff plus long per-kind Timeout methods could therefore bring sync and push rescue back near 1 hour.
  • Costs of that design: a Timeout per kind kept above that kind's budget, and the sweep, which has no budget of its own, would be rescued the moment its deadline passes while its 15-minute detached recovery can still run.

Owner: @ThomasK33.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

Lingua principale
Go
Stelle
0
Fork
0
Merge medio
6h 16m
PR unite (30g)
243

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di coder/openstorage

Tutte le issue di coder/openstorage

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.