jobs: follow-ups to the River job deadline fix (#632): Close without a drain, sweep without a pass cap, slower sync rescue

未关闭
  • 哪里不对？
#636 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
重构
描述清晰度
基本清楚
活跃度
活跃
技术栈
go
领域
backend, devops

调研方向

Start with the referenced shutdown paths in internal/jobs/river.go and cmd/openstorage/main.go, then trace the sweep in internal/jobs/sweep.go and blob.WithRenewedCover in internal/blob/blob.go. The issue lists three separate follow-ups but no selected scope or acceptance criteria; clarify which behavior to address before changing code. Done should include a focused fix and coverage for the affected shutdown, sweep, or rescue behavior.

由索引模型根据 Issue 内容生成。

描述

storage-gc tracking

The fix for #632 gives every River job a 2-hour deadline and rescues a crash-orphaned job after 2 h 30 min. Two limits stay. I found both by reading the code. I did not run either at runtime.

1. River.Close can wait up to the job deadline on serve's startup-failure path

  • River.Close calls client.Stop(context.Background()), which waits for every in-flight job (internal/jobs/river.go, Close).
  • Two kinds of exit reach Close without DrainSyncRuns first, while the run-on-start sweep or another job can be running:
    • serve's wiring fails after NewRiver started the client: cmd/openstorage/wire.go calls queueCleanup().
    • runServe returns early after defer cleanup() (cmd/openstorage/main.go): a setup error such as NewGitRateLimit, the HTTP listener stopping (serveErr), or the metrics listener stopping (metricsErr).
  • Before #632 the 1-minute deadline ended such a job early. Now the job can run until its own budget or the 2-hour deadline ends, plus the sweep's 15-minute detached recovery. So such an exit can take that long.
  • Signal shutdown is not affected: DrainSyncRuns cancels the remaining jobs when its context ends (cmd/openstorage/main.go, the drain after the termination signal).
  • Fix idea: in Close, stop with a bounded context, then StopAndCancel.

2. The orphan sweep has no pass cap and no resume cursor

  • In Postgres mode the 2-hour River deadline is now the only bound on one sweep pass (SweepSchedule.runOnce in internal/jobs/sweep.go runs the sweep and then the repo purge on the job context). Local mode has no bound.
  • A fleet whose sweep needs more than 2 hours never finishes a pass in Postgres mode, and the purge after it never runs. Each pass restarts from the beginning.
  • A sweep stuck in the cover-handoff retry loop of blob.WithRenewedCover (internal/blob/blob.go, the final loop, which retries while renewals succeed by design) can also outlive the 2 h 30 min rescue horizon. The next sweep then runs beside it, which the sweep tolerates (internal/jobs/sweep.go, "A concurrent sweep purged this entry first").
  • Trigger: the first fleet whose sweep takes more than about 1 hour, or a resumable sweep design.

3. Crash rescue for sync and push now waits 2 h 30 min instead of 1 hour

  • After a crash, an orphaned upstream_sync or upstream_push row stays running until River rescues it. It is unique including the running state (internal/jobs/river.go, syncArgs and pushArgs InsertOpts), so new requests for that repo deduplicate against it. They stay marked, so the rescued run catches them up, but only after 2 h 30 min.
  • River skips a job until both the client cutoff (RescueStuckJobsAfter) and the kind's own timeout have passed (vendor/github.com/riverqueue/river/internal/maintenance/job_rescuer.go, the timeout check in the retry decision). A short client cutoff plus long per-kind Timeout methods could therefore bring sync and push rescue back near 1 hour.
  • Costs of that design: a Timeout per kind kept above that kind's budget, and the sweep, which has no budget of its own, would be rescued the moment its deadline passes while its 15-minute detached recovery can still run.

Owner: @ThomasK33.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

主要语言
Go
星标
0
派生
0
平均合并
6 小时 16 分钟
30 天内合并 PR
243

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

coder/openstorage 的其他 Issue

查看 coder/openstorage 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。