[Enhancement] Durable workflow suspend and resume with persisted execution state (PAI-TS-001)

Aperta
#5,429 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
sqlite, typescript
Ambito
api, backend, cli, databases

Direzione di ricerca

Start in src/praisonai-ts/src/workflows/index.ts to see how AgentFlow steps run and where WorkflowContext lives in memory, then read src/praisonai-ts/src/db/ for the existing SQLite session patterns. The design calls for a new WorkflowStorageAdapter with suspend and resume, so this is design work rather than a first task. Done means the WF-SUSP and WF-UNIT tests in the issue pass against a real adapter.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug claude documentation javascript security

[Enhancement] Durable workflow suspend and resume with persisted execution state

Metadata

Field Value
Repository MervinPraison/PraisonAI
Package praisonai (npm) / monorepo path src/praisonai-ts
Component workflows, session storage, approval
Labels enhancement, workflows, human-in-the-loop, storage, praisonai-ts
Priority P1
Estimated effort 3–5 engineer-weeks

Executive summary

Production agent workflows often require pausing execution until a human approves an action, until external data arrives, or until a compliance review completes. The TypeScript SDK supports workflow orchestration, branching, parallelism, and YAML-defined flows, but it does not yet provide a first-class, storage-backed suspend and resume contract at the workflow step level. Tool-level restart safety exists; approval modules exist; however, there is no durable checkpoint that survives process restarts and no canonical API for operators to resume a workflow run days later.

This issue defines the product requirement, public API, persistence model, security constraints, and exhaustive test matrix needed to ship durable suspend/resume for AgentFlow and related workflow patterns.

Interactive triage (start here)

Use this decision tree to confirm the gap applies to your report:

flowchart TD
  A[Workflow needs human approval mid-run?] -->|Yes| B[Process may restart before approval?]
  A -->|No| Z[Close as not applicable]
  B -->|Yes| C{After restart, can you resume same run id?}
  B -->|No| Z
  C -->|No| D[File under this issue]
  C -->|Yes| Z
  D --> E[Attach repro log from section Terminal evidence]
Contributor checklist (click to expand)
  • I reproduced loss of workflow context after process exit
  • I confirmed AgentFlow runs without WorkflowStorageAdapter
  • I read Expected vs actual below
  • I reviewed Suggested code changes and linked modules

Architecture — current (in-memory)

flowchart LR
  subgraph Process
    AF[AgentFlow]
    WC[WorkflowContext in RAM]
    ST[Steps / patterns]
    AF --> ST --> WC
  end
  APP[approval module] -. agent-level only .-> AF
  TOOL[FunctionTool resumed flag] -. tool-level only .-> ST
  WC -->|process exit| LOST[(State lost)]
Layer Module Responsibility Survives restart?
Orchestration src/praisonai-ts/src/workflows/index.ts Step execution No
Context WorkflowContext Key/value per run No
Human gate src/praisonai-ts/src/approval/ Approve/deny actions Partial
Tool idempotency src/praisonai-ts/src/tools/decorator.ts resumed + isRestartSafe Per tool call only
Persistence src/praisonai-ts/src/db/ SQLite sessions Not wired to workflows

Architecture — proposed (durable)

sequenceDiagram
  participant Dev as Application
  participant Flow as AgentFlow
  participant Store as WorkflowStorageAdapter
  participant Op as Operator CLI

  Dev->>Flow: run(input, { storage })
  Flow->>Store: createRun + checkpoints each step
  Flow->>Flow: step requires approval
  Flow->>Store: saveCheckpoint(status=suspended)
  Note over Flow: process may stop here
  Op->>Flow: resume(runId, { approved: true })
  Flow->>Store: loadCheckpoint
  Flow->>Flow: continue from step_index
  Flow->>Store: mark completed
Component New / changed Storage
WorkflowStorageAdapter New interface Pluggable
SqliteWorkflowStorage New Reuse db() patterns
AgentFlow.run / resume Extend Optional adapter
AgentOS routes New HTTP resume

Expected vs actual behaviour

# Scenario Expected (product) Actual (today) Severity
1 Workflow suspends for approval Run id stable; status suspended in store Context only in memory High
2 Server restart during suspend Operator resumes with same run id Run lost; must restart workflow High
3 Parallel branch waits on human Only blocked branch suspended Entire flow blocked or undefined Medium
4 YAML workflow without storage Same as today (no regression) Works in-memory OK
5 CLI lists suspended runs JSON table of run ids No command Medium
6 Audit who resumed resumed_by persisted Not available Medium

Terminal evidence (reproduction)

Evidence A — workflow unit tests pass without persistence (expected today)
> jest tests/unit/workflows

PASS tests/unit/workflows/patterns.test.ts
PASS tests/unit/workflows/workflow.test.ts
PASS tests/unit/workflows/yaml-parser.test.ts

Test Suites: 13 passed, 13 total
Tests:       230 passed, 230 total

Interpretation: orchestration logic is sound; gap is durability, not step semantics.

Evidence B — conceptual repro script (suspend gap)

Run after building the SDK:

> node -e "
  const { AgentFlow } = require('./dist/workflows');
  const flow = new AgentFlow('approval-demo');
  flow.addStep({ name: 'need-approval', execute: async (ctx) => {
    ctx.set('pending', true);

> **Note:** Full diagrams, code patches, and terminal evidence live in `rich_sections.py` for this issue; this draft stays ~300 lines for review.

---

## Regression guardrails

| Check | Command |
|-------|---------|
| Unit tests | `npm test -- tests/unit/workflows` |
| Build | `npm run build` |

**Related:** `PAI-TS-001` cross-links other PAI-TS platform drafts in the same milestone.

## Problem statement

Long-running workflows fail or block when:
- A step requires human approval and the server process restarts.
- An external webhook will arrive asynchronously.
- Operators need to inspect intermediate state before continuing.

Today, in-memory workflow context is lost on crash. Partial mitigations (tool `resumed` flags, approval helpers) do not replace a persisted run record.

## Goals

1. Introduce `WorkflowRun` identity durable across restarts.
2. Allow any step to call `suspend(reason, payload)` and persist snapshot.
3. Allow `resume(runId, input)` from CLI, HTTP API, and programmatic SDK.
4. Integrate with existing SQLite adapter first; define port for Postgres.
5. Maintain backward compatibility: workflows without storage behave as today.

## Non-goals

- Distributed workflow engine across multiple regions in v1.
- Visual workflow editor (tracked separately).
- Replacing agent chat memory systems.
- Guaranteeing exactly-once side effects for non-idempotent tools without user configuration.

## Personas

| Persona | Need |
|---------|------|
| Platform engineer | Operate workflows in production with restart safety |
| Application developer | Simple API to pause for approval |
| Compliance reviewer | Audit who resumed and with what input |
| SRE | Metrics on suspended run age and resume latency |

## Persistence schema (logical)

| Column / field | Type | Description |
|----------------|------|-------------|
| run_id | UUID | Primary key |
| workflow_name | string | Registered flow name |
| status | enum | running/suspended/completed/failed |
| step_index | int | Current step pointer |
| context_json | json | Serializable workflow context |
| suspend_reason | string | Optional |
| suspend_payload | json | Operator-visible data |
| created_at | timestamp | UTC |
| updated_at | timestamp | UTC |
| resumed_by | string | Actor id from auth context |

## Security requirements

- SR-001: All resume operations require authenticated actor id in audit log.
- SR-002: Checkpoint payloads must not store raw secrets; use references only.
- SR-003: Cross-tenant run id access returns 404 without leaking existence.
- SR-004: Rate-limit resume API to prevent brute-force run id guessing.
- SR-005: Optional encryption at rest for context_json when enterprise flag set.

## Functional requirements

- FR-001: Persist checkpoint within 500ms of suspend call (p95).
- FR-002: Expose runId to application after first suspend.
- FR-003: Resume accepts optional operator input merged into context.
- FR-004: Failed step may suspend or fail based on workflow policy.
- FR-005: listSuspended returns paginated results sorted by suspendedAt desc.
- FR-006: Storage adapter interface documented and mockable in tests.

## Non-functional requirements

- NFR-001: Resume latency p95 under 2s for checkpoints under 1MB.
- NFR-002: SQLite adapter supports 10k suspended runs without full table scan on list.
- NFR-003: Checkpoint write must not block event loop >100ms without async offload.

## CLI requirements

- CLI-001: `praisonai-ts workflow list --suspended` returns JSON table.
- CLI-002: `praisonai-ts workflow resume <runId> --input file.json`.
- CLI-003: Exit code 2 when run not found or not suspended.

## HTTP API requirements (AgentOS)

- API-001: POST /workflow-runs/{id}/resume accepts JSON body.
- API-002: GET /workflow-runs/{id} returns status and suspend metadata.
- API-003: GET /workflow-runs?suspended=true supports cursor pagination.

## Integration test matrix

| ID | Scenario | Preconditions | Steps | Expected |
|----|----------|---------------|-------|----------|
| WF-SUSP-001 | Suspend mid-step then resume after simulated process restart | SDK built; storage adapter configured | Execute steps in test spec 1 | State consistent; typed errors; no data loss |
| WF-SUSP-002 | Reject resume when run id unknown | SDK built; storage adapter configured | Execute steps in test spec 2 | State consistent; typed errors; no data loss |
| WF-SUSP-003 | Parallel branch A suspended while branch B completes | SDK built; storage adapter configured | Execute steps in test spec 3 | State consistent; typed errors; no data loss |
| WF-SUSP-004 | Include pattern nested inside loop preserves step pointer | SDK built; storage adapter configured | Execute steps in test spec 4 | State consistent; typed errors; no data loss |
| WF-SUSP-005 | Checkpoint size near 1MB still loads under NFR budget | SDK built; storage adapter configured | Execute steps in test spec 5 | State consistent; typed errors; no data loss |
| WF-SUSP-006 | Concurrent resume requests are idempotent | SDK built; storage adapter configured | Execute steps in test spec 6 | State consistent; typed errors; no data loss |
| WF-SUSP-007 | Cancel suspended run transitions to cancelled status | SDK built; storage adapter configured | Execute steps in test spec 7 | State consistent; typed errors; no data loss |
| WF-SUSP-008 | YAML workflow without storage adapter unchanged behaviour | SDK built; storage adapter configured | Execute steps in test spec 8 | State consistent; typed errors; no data loss |

## Unit test matrix

| ID | Scenario | Preconditions | Steps | Expected |
|----|----------|---------------|-------|----------|
| WF-UNIT-001 | WorkflowContext JSON round-trip preserves keys | SDK built; storage adapter configured | Execute steps in test spec 1 | State consistent; typed errors; no data loss |
| WF-UNIT-002 | Storage adapter createRun + saveCheckpoint + loadCheckpoint | SDK built; storage adapter configured | Execute steps in test spec 2 | State consistent; typed errors; no data loss |
| WF-UNIT-003 | suspend() writes status suspended and reason | SDK built; storage adapter configured | Execute steps in test spec 3 | State consistent; typed errors; no data loss |
| WF-UNIT-004 | resume() rejects completed runs | SDK built; storage adapter configured | Execute steps in test spec 4 | State consistent; typed errors; no data loss |
| WF-UNIT-005 | listSuspended filters by workflow name | SDK built; storage adapter configured | Execute steps in test spec 5 | State consistent; typed errors; no data loss |
| WF-UNIT-006 | Serialization strips non-JSON values with clear error | SDK built; storage adapter configured | Execute steps in test spec 6 | State consistent; typed errors; no data loss |

## Migration notes

- MIG-001: Existing YAML workflows require no changes when storage unset.
- MIG-002: Opt-in: pass storage adapter in AgentFlow.run options only.
- MIG-003: v1 SQLite schema versioned; migration script for future columns.

## Documentation outline

- DOC-001: Sequence diagram for suspend/resume (Mintlify).
- DOC-002: Operator guide: list and resume suspended runs.
- DOC-003: Adapter implementation guide for Postgres.
- DOC-004: Security: authn/z on resume endpoints.
- DOC-005: Troubleshooting: stale locks and corrupted checkpoints.

## Proposed public API (summary)

`WorkflowStorageAdapter` with createRun, saveCheckpoint, loadCheckpoint, listSuspended.
`AgentFlow.run(input, { storage })` and `AgentFlow.resume(runId, input)`.
See **Architecture — proposed** in the triage section for sequence diagram.

## Acceptance criteria

- [ ] Suspend survives process kill and resume completes successfully.
- [ ] Parallel branch suspend resumes only affected branch.
- [ ] Unit tests cover serialization round-trip for WorkflowContext.
- [ ] Documentation published under docs/workflows/durable-execution.md.
- [ ] No breaking change for flows not passing storage adapter.

## Rollout plan

Phase 1: SQLite adapter + in-process API.
Phase 2: AgentOS routes + CLI.
Phase 3: Postgres adapter + migration tool.
Phase 4: Metrics and alerting hooks.

## Open questions

- OQ-001: Should nested Include patterns inherit the parent run id or allocate a child run id?
- OQ-002: How should parallel branches checkpoint: one shared snapshot or one per branch?
- OQ-003: What is the retention policy for suspended runs (TTL, manual purge, compliance hold)?
- OQ-004: Should resume require the same authenticated actor who created the run?
- OQ-005: How do we version checkpoint JSON when WorkflowContext schema evolves?
Lingua principale
Python
Stelle
9.3k
Fork
1.5k
Merge medio
1g 5h
PR unite (30g)
520

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di MervinPraison/PraisonAI

Tutte le issue di MervinPraison/PraisonAI

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.