[Feature]: A checkpoint that stores an actor without suspending it

Open
  • Is something wrong?
#3,299 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
backend, distributed-systems

Research direction

Read the existing SuspendActor flow and its DurabilityMode handling, then inspect how suspended snapshots are uploaded to object storage. The issue proposes CheckpointActor or a SuspendActor option that captures a snapshot and resumes the actor while upload continues. Done means a busy actor can gain a new durable snapshot without being suspended; the issue names no files or tests, so first locate those entry points and related tests.

Written by the indexing model from the issue text.

Description

area/api area/api-machinery area/node area/storage kind/feature
What we'd like

A way to store a durable snapshot of an actor while it keeps running: a checkpoint that doesn't suspend it, uploaded to object storage the way a suspend's snapshot is.

Why

We run Substrate v0.6.0 on GKE as the sandbox for agent sessions: each session is an actor, suspended when it goes idle and resumed at the next message. Today an actor's state is stored only by suspending it (SuspendActor, with DurabilityMode synchronous or asynchronous). DrainWorker deliberately leaves bound actors alone, so draining a node stores nothing.

So an actor that stays busy is never stored. An agent turn can run for 40 minutes, or a user can send messages a minute apart for an hour. If GKE replaces the node during that stretch (a routine node upgrade replaces every node in turn), the actor goes back to its last suspend, and everything since is lost.

Suspending the actor between requests isn't a substitute:

  • a suspend in the middle of a request cancels it (#1572), and a long turn has no gap between requests;
  • suspending and resuming a chatty actor every few minutes charges each of those messages a resume.
What it would look like

Something like a CheckpointActor (or a SuspendActor option that resumes the actor once the snapshot is captured on the node), whose snapshot then uploads in the background and becomes the actor's latest durable one. A client could then store a busy actor on a schedule, or between turns, and bound what a node replacement costs.

Related: #2470 (resuming an actor whose node is gone), which is the other half of surviving a node replacement.

Dominant language
Go
Stars
5.9k
Forks
637
Avg merge
2d 1h
Merged PRs (30d)
523

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.