Recover abandoned started agent runs after runtime shutdown

Open
#4,633 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
typescript
Domain
backend

Research direction

Start from the runtime handler catch path and the existing started-run failure handling described in the issue. Trace how started runs are persisted and how runtime startup occurs, then add recovery tests for an interrupted run. Done means restart recovery marks stale runs terminal with an explanatory error without failing active long-running runs.

Written by the indexing model from the issue text.

Description

Problem

The UI shows "Thinking" while the latest agent run remains . We fixed one runtime failure path where a handler throws after starting a run, but hard process death / laptop sleep / runtime shutdown can still leave a started run without any terminal update because the process never gets a chance to run catch/finally cleanup.

Proposed second layer

Add recovery semantics for abandoned started runs, e.g. one of:

  • a run lease / heartbeat renewed while a run is active, plus a janitor/startup sweep that marks stale started runs failed, or
  • a simpler startup recovery sweep that marks old runs failed with a clear interrupted/abandoned error.

Acceptance criteria

  • A runtime restart after an interrupted run does not leave the UI stuck on Thinking indefinitely.
  • Stale started runs are marked terminal ( or equivalent) with an explanatory / error message.
  • Long-running active runs are not incorrectly failed if heartbeat/lease support is implemented.
  • Add tests covering an abandoned started run recovery path.

Context

Current PR addresses handler exceptions after a run starts by failing the newly-started run in the handler catch path. This issue tracks crash/shutdown recovery where no in-process catch/finally can run.

Dominant language
TypeScript
Stars
10.4k
Forks
399
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from electric-sql/electric

All issues in electric-sql/electric

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.