jobs: follow-ups to the River job deadline fix (#632): Close without a drain, sweep without a pass cap, slower sync rescue

Đang mở
  • Có gì không ổn?
#636 3 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tái cấu trúc
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go
Lĩnh vực
backend, devops

Hướng nghiên cứu

Start with the referenced shutdown paths in internal/jobs/river.go and cmd/openstorage/main.go, then trace the sweep in internal/jobs/sweep.go and blob.WithRenewedCover in internal/blob/blob.go. The issue lists three separate follow-ups but no selected scope or acceptance criteria; clarify which behavior to address before changing code. Done should include a focused fix and coverage for the affected shutdown, sweep, or rescue behavior.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

storage-gc tracking

The fix for #632 gives every River job a 2-hour deadline and rescues a crash-orphaned job after 2 h 30 min. Two limits stay. I found both by reading the code. I did not run either at runtime.

1. River.Close can wait up to the job deadline on serve's startup-failure path

  • River.Close calls client.Stop(context.Background()), which waits for every in-flight job (internal/jobs/river.go, Close).
  • Two kinds of exit reach Close without DrainSyncRuns first, while the run-on-start sweep or another job can be running:
    • serve's wiring fails after NewRiver started the client: cmd/openstorage/wire.go calls queueCleanup().
    • runServe returns early after defer cleanup() (cmd/openstorage/main.go): a setup error such as NewGitRateLimit, the HTTP listener stopping (serveErr), or the metrics listener stopping (metricsErr).
  • Before #632 the 1-minute deadline ended such a job early. Now the job can run until its own budget or the 2-hour deadline ends, plus the sweep's 15-minute detached recovery. So such an exit can take that long.
  • Signal shutdown is not affected: DrainSyncRuns cancels the remaining jobs when its context ends (cmd/openstorage/main.go, the drain after the termination signal).
  • Fix idea: in Close, stop with a bounded context, then StopAndCancel.

2. The orphan sweep has no pass cap and no resume cursor

  • In Postgres mode the 2-hour River deadline is now the only bound on one sweep pass (SweepSchedule.runOnce in internal/jobs/sweep.go runs the sweep and then the repo purge on the job context). Local mode has no bound.
  • A fleet whose sweep needs more than 2 hours never finishes a pass in Postgres mode, and the purge after it never runs. Each pass restarts from the beginning.
  • A sweep stuck in the cover-handoff retry loop of blob.WithRenewedCover (internal/blob/blob.go, the final loop, which retries while renewals succeed by design) can also outlive the 2 h 30 min rescue horizon. The next sweep then runs beside it, which the sweep tolerates (internal/jobs/sweep.go, "A concurrent sweep purged this entry first").
  • Trigger: the first fleet whose sweep takes more than about 1 hour, or a resumable sweep design.

3. Crash rescue for sync and push now waits 2 h 30 min instead of 1 hour

  • After a crash, an orphaned upstream_sync or upstream_push row stays running until River rescues it. It is unique including the running state (internal/jobs/river.go, syncArgs and pushArgs InsertOpts), so new requests for that repo deduplicate against it. They stay marked, so the rescued run catches them up, but only after 2 h 30 min.
  • River skips a job until both the client cutoff (RescueStuckJobsAfter) and the kind's own timeout have passed (vendor/github.com/riverqueue/river/internal/maintenance/job_rescuer.go, the timeout check in the retry decision). A short client cutoff plus long per-kind Timeout methods could therefore bring sync and push rescue back near 1 hour.
  • Costs of that design: a Timeout per kind kept above that kind's budget, and the sweep, which has no budget of its own, would be rescued the moment its deadline passes while its 15-minute detached recovery can still run.

Owner: @ThomasK33.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

Ngôn ngữ chính
Go
Star
0
Fork
0
Merge trung bình
6 giờ 16 phút
Pull request đã merge (30 ngày)
243

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của coder/openstorage

Tất cả issue của coder/openstorage

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.