Exhausted authority preactivation teardown stays running and cannot be recovered

Open
  • Is something wrong?
#3,617 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start in src/kdive/services/systems/authority_owned.py around enqueue_preactivation_teardown, where a prior job is returned unless its state is FAILED, and in src/kdive/jobs/worker.py _finalize_handler, which leaves a marked teardown in running without a terminal receipt. Read the reconciler and the runbook sections in docs/operating/runbooks/live-testing.md before choosing where the exhausted state becomes terminal. Done means a test drives a teardown to max_attempts without a receipt and then recovers it once the authority is healthy, and the runbook Limits section states the bounded behaviour.

Written by the indexing model from the issue text.

Description

area:provisioning effort:M priority:P2 risk:daytime-only status:ready type:bug

Problem

When an authority-owned System's preactivation teardown uses up its attempts without a terminal receipt, nothing can recover it. The teardown job stays running with attempt equal to max_attempts (3) and a lapsed lease. No path terminalizes it:

  • systems.teardown and ops.force_teardown return the same job;
  • jobs.cancel refuses it;
  • ops.force_release is fenced;
  • the dedup only recycles a failed row, and this row never becomes failed.

Fixing the underlying cause afterwards does not help, because no new attempt is ever scheduled. The System and its Allocation stay stranded.

Observed in live cells C1 and C2 of #3584. The Exhausted authority System provision runbook also implies that the teardown retries forever, which is not true.

Evidence

  • src/kdive/services/systems/authority_owned.py:275-286: enqueue_preactivation_teardown returns a prior job as-is when prior.state is not JobState.FAILED (if prior is not None and prior.state is not JobState.FAILED: job = prior). Only a failed row is recycled.
  • src/kdive/jobs/worker.py:535-549: in _finalize_handler, a job that carries an authority System marker and fails without a terminal receipt is logged (marked authority System job %s failed without a terminal receipt) and the method returns False. The job row is not failed, so it stays running.
  • src/kdive/mcp/tools/jobs.py:301-312: jobs.cancel refuses the job with reason authority_system_preactivation_teardown_not_cancelable.
  • docs/operating/runbooks/live-testing.md:484-487 (from PR #3611): "After its last attempt, nothing restarts it. systems.teardown and ops.force_teardown return the same job, and jobs.cancel refuses it".
  • No reconciler path terminalizes an exhausted marked teardown that has no receipt. The provision job has such a path (the reconciler cancels it once the teardown supersedes it, live-testing.md:504-508), but the teardown itself does not.
  • docs/operating/runbooks/exhausted-authority-system-provision.md:46-49, "Limits": "Restore the authority service and let the teardown retry." This implies unbounded retry and does not mention the max_attempts stop.

Expected

  • Once the cause is fixed, an operator can recover an exhausted preactivation teardown through a supported path. Examples: the teardown is moved to failed so the existing operator recycle (#3530 policy) admits it, or an admin re-arm. No lab rollback is needed, and no manual row edits.
  • The runbook's "Limits" section states the max_attempts stop and links to the #3611 subsection, Recovering an allocation stranded by a failed authority-owned provision.

Proposed approach

  1. Choose where the exhausted state becomes terminal. Two candidates:
    • the reconciler fails a marked preactivation teardown whose attempt == max_attempts and whose lease has lapsed with no receipt;
    • the operator recycle in enqueue_preactivation_teardown admits that row as well as failed.
      Either way, keep the invariant that release stays fenced until the authority's absence receipt exists. Record the decision as an amendment if it changes the teardown state machine of an accepted ADR.
  2. Add tests that drive a teardown to exhaustion without a receipt and then recover it once the authority is healthy.
  3. Docs fix, in the same PR: in exhausted-authority-system-provision.md "Limits", replace "let the teardown retry" with the bounded behaviour (three attempts, then the recovery path) and link to the #3611 subsection.

Related: the binding-mismatch dead end filed alongside this issue. That case cannot be recovered by fixing the cause and needs its own exit.

Dominant language
Python
Stars
0
Forks
0
Avg merge
9h 13m
Merged PRs (30d)
197

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from randomparity/kdive

All issues in randomparity/kdive

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.