Stranded runs.boot leaves its exhausted authority-marked boot job running until jobs.cancel

Open
  • Is something wrong?
#3,596 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
postgresql, python, sql
Domain
backend, database

Research direction

Start with dead_letter_unowned_external_boot_jobs in src/kdive/db/schema/0165_bound_acknowledged_retry_grant.sql and repair_abandoned_jobs in src/kdive/reconciler/repairs/jobs.py to see why an exhausted authority-marked boot job stays running. Then read the ops.recover_stranded_activation settle path and the runbook at docs/operating/runbooks/stranded-activation-recovery.md. Done means the job is ended by the chosen owner, the job_live refusal and runbook step 2 are updated, and the real-database test no longer cancels the job by hand. The owner decision is still open in the thread, so agree on it first.

Written by the indexing model from the issue text.

Description

area:core-platform effort:M priority:P3 risk:daytime-only status:ready type:bug

Problem

A runs.boot that strands a fixed-worker System leaves its exhausted, authority-marked boot job running after its last attempt. The reconciler's dead_letter_unowned_external_boot_jobs skips it while its authority row is allocating or current, so nothing ends the job. settle_stranded_external_boot_activation then refuses job_live until an operator runs jobs.cancel by hand. The stranded-activation recovery (ADR-0816, ops.recover_stranded_activation) therefore has a manual step that the tool and the reconciler both leave to the operator.

Evidence

  • src/kdive/db/schema/0165_bound_acknowledged_retry_grant.sql:36- (dead_letter_unowned_external_boot_jobs): for a running boot job with attempt >= max_attempts, an expired lease and an authority marker, an allocating/current authority row reaches CONTINUE WHEN NOT EXISTS (... acknowledged_retry_consumptions ...). Without a consumption row for the claimed attempt the job is skipped and stays running. This is the latest definition (git grep finds no later redefinition).
  • src/kdive/reconciler/repairs/jobs.py (repair_abandoned_jobs): the generic sweep excludes authority-marked jobs by design, so no other lane ends it.
  • ADR-0816 (settlement): the shape requires "no queued or running authority-marked job"; the function refuses job_live otherwise.
  • docs/operating/runbooks/stranded-activation-recovery.md, Recover step 2: documents the manual jobs.cancel when attempt equals max_attempts and the lease has lapsed.
  • Reproduced in the live proof for #3552: settlement refused job_live until the exhausted boot job was cancelled.

Expected

Decide which owner ends the exhausted job for the stranded shape, so the runbook step is not needed:

  • the reconciler, when no work remains for it (the authority never acknowledged a mutation and the activation is preparing with no ownership row), or
  • ops.recover_stranded_activation settle, which cancels or fails an exhausted job with a lapsed lease as part of the audited settlement.

Either way the Run is compensated as the existing dead-letter path does, and a job a worker still holds (lease live, or attempts remaining) is never touched.

Proposed approach

Prefer the tool: settle already runs under the System lock with platform_admin and a recorded reason, which is the authority the reconciler lacks for an authority-marked job. Before refusing job_live, end a boot job of the System whose attempt >= max_attempts and whose lease has lapsed (same predicate as the runbook), through the existing job-cancel service, and audit it. Keep refusing job_live for any other live job. Update the runbook step 2 and the job_live refusal row, and extend the real-database arm that currently cancels the job by hand. If the reconciler is chosen instead, the change is a new migration amending the CONTINUE condition for the stranded shape; weigh that against the 0165 bound on acknowledged retries before choosing it.

Provenance

Follow-up from campaign 4c2c8dc9e59d-7f9a87f5-2251-458a-bb30-04fbb2d4b05c, PR #3595.

Dominant language
Python
Stars
0
Forks
0
Avg merge
9h 13m
Merged PRs (30d)
197

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from randomparity/kdive

All issues in randomparity/kdive

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.