Ops rules: Resque silent job loss — mid-ramp dispatch ban + end-of-run sweep

Open Beginner friendly

#119 opened on Aug 24, 2026 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 2/5
Estimated time: 1-3 hours
Newbie friendliness: 74/100

Issue type: Documentation
Clarity: Mostly clear
Activity status: Active
Tech stack: helm, kubernetes, mongodb, redis
Domain: databases, devops, documentation, infrastructure

Research direction

Start with HANDOFF-SCALE-TO-9000.md §R7 and compare its three rules with the issue's listed MongoDB and fleet-operation caveats. Add a runbook section or standalone document covering convergence before dispatch, post-recycle redispatch, and the end-of-run sweep; document the cursor behavior and Mongo access details, with sweep tooling optional.

Written by the indexing model from the issue text.

Description

Why

HANDOFF-SCALE-TO-9000.md §R7: jobs die silently when worker children die mid-run (verified again 2026-08-24). This is a process contract, not a bug to patch today — but it's currently tribal knowledge in a handoff doc.

The rules (from the handoff, verified)

Dispatch/re-dispatch ONLY after fleet fully converged: readyReplicas == desired AND all mounts ls clean. Mid-ramp dispatch caused the 2026-08-24 orphan wave.
After ANY mass recycle: re-dispatch dps with status: started (their children died). queued jobs survive in redis and resume on their own.
End-of-run sweep: any dp not completed + completed normal gets one submit_simulation once the fleet is quiet.

Caveats / gotchas to document

Beware mongoid cursor weirdness: updating while iterating re-emits updated docs (#each) — harmless duplicates, app skips completed-normal, but document it so nobody "fixes" it into a livelock.
Mongo access: exec into deploy/db, mongosh -u openstudio -p <pass> --authenticationDatabase admin os_docker, collection data_points.

Deliverable

Runbook section (or standalone doc) covering the three rules
Optionally: scripted end-of-run sweep tooling so rule 3 isn't hand-run mongosh

Dominant language: Go Template
Stars: 12
Forks: 24
PR merge metrics: No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

Batched `completeJobs`/`failJobs` are never retried: the retry loop in `batch()` exits after the first failure Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

graphile/worker#636 · Opened Sep 21, 2026
Test Failure: com.ibm.ws.jdbc.fat.sqlserver.SQLServerTest J2CA0079E Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

OpenLiberty/open-liberty#35839 · Opened Sep 21, 2026
hash join v2: 'unsupported column type for pre-alloc 225' when joining on a VECTOR column Open

affects-9.0 found-by-ai may-affects-25.10 may-affects-26.3 may-affects-26.9 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major sig/execution type/bug

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

pingcap/tidb#71473 · Opened Sep 21, 2026
JSON_VALUE fails to parse when a RETURNING clause is present but the optional ON EMPTY/ON ERROR clause is omitted Open

Difficulty 1/5 Under an hour Newbie friendliness 92/100

antlr/grammars-v4#5014 · 1 comment · Opened Sep 21, 2026
readBinary(std::vector) throws a bare Poco::Exception (POCO_EXCEPTION code 1000) for too-large sizes instead of TOO_LARGE_ARRAY_SIZE Open

comp-query-execution fuzz

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

ClickHouse/ClickHouse#121303 · 3 comments · Opened Sep 21, 2026

More Databases issues