[feat][GraphWorkflow] Add checkpoint_dir parameter for mid-graph persist and resume

Summary

Long-running GraphWorkflow executions have no fault tolerance. If an agent fails or the process crashes mid-graph, the entire workflow must restart from scratch — including re-running all previously completed agents.

Proposed Enhancement

Add a checkpoint_dir parameter that automatically persists prev_outputs to disk after each layer completes, and resumes from the last successful checkpoint on restart:

GraphWorkflow(
    ...,
    checkpoint_dir="./checkpoints/run_abc123",
)

Checkpoint behavior

• After each layer completes, serialize prev_outputs to {checkpoint_dir}/layer_{idx}.json.
• On run(), check if a checkpoint exists for the current task. If so, skip already-completed layers and load their outputs from disk.
• Checkpoint files are keyed by (task_hash, layer_idx) to avoid collisions across different tasks.

checkpoint_path = Path(checkpoint_dir) / f"{hash(task)}_layer_{layer_idx}.json"
if checkpoint_path.exists():
    prev_outputs.update(json.loads(checkpoint_path.read_text()))
    continue  # skip re-executing this layer

Cleanup

Add a clear_checkpoints(task: str) method to delete checkpoint files after successful completion.

Use Cases

• Multi-hour agentic pipelines where any single agent may time out or hit API limits.
• Cost savings — avoid paying for re-running expensive LLM calls on agents that already succeeded.
• Debugging — inspect the exact output of each layer independently.

File

swarms/structs/graph_workflow.py — run() method, layer execution loop (~line 1750)