jobs: follow-ups to the River job deadline fix (#632): Close without a drain, sweep without a pass cap, slower sync rescue

Aberta
  • Algo errado?
#636 3 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Refatoração
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
go
Domínio
backend, devops

Direção de pesquisa

Start with the referenced shutdown paths in internal/jobs/river.go and cmd/openstorage/main.go, then trace the sweep in internal/jobs/sweep.go and blob.WithRenewedCover in internal/blob/blob.go. The issue lists three separate follow-ups but no selected scope or acceptance criteria; clarify which behavior to address before changing code. Done should include a focused fix and coverage for the affected shutdown, sweep, or rescue behavior.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

storage-gc tracking

The fix for #632 gives every River job a 2-hour deadline and rescues a crash-orphaned job after 2 h 30 min. Two limits stay. I found both by reading the code. I did not run either at runtime.

1. River.Close can wait up to the job deadline on serve's startup-failure path

  • River.Close calls client.Stop(context.Background()), which waits for every in-flight job (internal/jobs/river.go, Close).
  • Two kinds of exit reach Close without DrainSyncRuns first, while the run-on-start sweep or another job can be running:
    • serve's wiring fails after NewRiver started the client: cmd/openstorage/wire.go calls queueCleanup().
    • runServe returns early after defer cleanup() (cmd/openstorage/main.go): a setup error such as NewGitRateLimit, the HTTP listener stopping (serveErr), or the metrics listener stopping (metricsErr).
  • Before #632 the 1-minute deadline ended such a job early. Now the job can run until its own budget or the 2-hour deadline ends, plus the sweep's 15-minute detached recovery. So such an exit can take that long.
  • Signal shutdown is not affected: DrainSyncRuns cancels the remaining jobs when its context ends (cmd/openstorage/main.go, the drain after the termination signal).
  • Fix idea: in Close, stop with a bounded context, then StopAndCancel.

2. The orphan sweep has no pass cap and no resume cursor

  • In Postgres mode the 2-hour River deadline is now the only bound on one sweep pass (SweepSchedule.runOnce in internal/jobs/sweep.go runs the sweep and then the repo purge on the job context). Local mode has no bound.
  • A fleet whose sweep needs more than 2 hours never finishes a pass in Postgres mode, and the purge after it never runs. Each pass restarts from the beginning.
  • A sweep stuck in the cover-handoff retry loop of blob.WithRenewedCover (internal/blob/blob.go, the final loop, which retries while renewals succeed by design) can also outlive the 2 h 30 min rescue horizon. The next sweep then runs beside it, which the sweep tolerates (internal/jobs/sweep.go, "A concurrent sweep purged this entry first").
  • Trigger: the first fleet whose sweep takes more than about 1 hour, or a resumable sweep design.

3. Crash rescue for sync and push now waits 2 h 30 min instead of 1 hour

  • After a crash, an orphaned upstream_sync or upstream_push row stays running until River rescues it. It is unique including the running state (internal/jobs/river.go, syncArgs and pushArgs InsertOpts), so new requests for that repo deduplicate against it. They stay marked, so the rescued run catches them up, but only after 2 h 30 min.
  • River skips a job until both the client cutoff (RescueStuckJobsAfter) and the kind's own timeout have passed (vendor/github.com/riverqueue/river/internal/maintenance/job_rescuer.go, the timeout check in the retry decision). A short client cutoff plus long per-kind Timeout methods could therefore bring sync and push rescue back near 1 hour.
  • Costs of that design: a Timeout per kind kept above that kind's budget, and the sweep, which has no budget of its own, would be rescued the moment its deadline passes while its 15-minute detached recovery can still run.

Owner: @ThomasK33.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

Linguagem predominante
Go
Estrelas
0
Forks
0
Merge médio
6h 16min
PRs com merge (30d)
243

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de coder/openstorage

Todas as issues de coder/openstorage

Issues semelhantes

Mais issues de Go

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.