RFC 004: Durable Step Values

This is a design RFC for step.effect and an EffectClient in @yieldstar/core, and it depends on RFCs 001, 002, 003 and 011 linked in the body. Read those linked RFCs and the existing step.run behavior before forming any view. Done means the design is discussed and accepted by maintainers, with the conformance table in the body turned into tests, not a code change.

Abstract

Replay is supposed to show workflow code exactly what the first run saw, but Yieldstar's serializer silently changes values on the way through. A Date comes back as a string, a Map as {}, and any object that looks like an error comes back as an Error, so the same code can take a different branch on replay.

We will solve this with one codec for every durable value, which accepts only values it can return unchanged and rejects the rest with a catchable error when they are first produced. First run and replay will see identical values. Errors will have their own tagged encoding that keeps their class, code and cause.

https://github.com/user-attachments/assets/960adf04-47fd-4a75-8cd0-262b5ca8f35c

Video transcript

Back to place-order. Its first step, reserve-stock, holds the last mug for order 42, and returns a reservation: an ID, and the time the hold expires, as a JavaScript Date. Yieldstar saves each step's result in the step's record, as JSON text. When the workflow runs again, the step returns the saved result, and does not call the warehouse again. Yieldstar calls this replay, and replay rests on one promise: the workflow sees exactly what the first run saw.

Problem 1. The serializer breaks that promise. On the first run, the expiry is a Date. Then place-order waits for payment, and the delivery ends. When the payment arrives, the next delivery replays reserve-stock from its record. But JSON has no dates, so the expiry comes back as a string. Place-order calls get time on a string, and throws a type error. So every order that waits for payment fails. Dates are not alone. A Map or a Set comes back as an empty object, an undefined property disappears, and a class instance loses its methods.

Problem 2 is errors, and here the serializer goes wrong both ways. Say the courier step throws a courier error (CourierError), and place-order catches that class to book a backup courier. On replay, the error comes back as a plain Error. The class is gone, so the check fails, and replay rethrows where the first run booked the backup. The other way round, a step that returns a plain object with a name, a message and a stack, say a crash report, gets an Error back on replay.

RFC 011 will replace the serializer with one codec: a single encoder and decoder for every value Yieldstar stores, from step results to store state and params. The codec will accept only durable values, the type from RFC 003: null, booleans, numbers, strings, and arrays and plain objects made of them. JSON returns these values unchanged. The codec will reject everything else.

The codec will reject a value when the step first produces it. If reserve-stock returns a Date, the step will throw a value error (WorkflowValueError: INVALID_STEP_VALUE) on the very first run. Yieldstar will record that error, so replay will throw the same one, and workflow code can catch it like any step error. This is the important shift: the bug moves from replay to the first run, so the developer sees it in their first test.

One more rule closes the gap. On the first run, workflow code will not get the object that the step function returned. It will get the decoded copy: the value encoded, then decoded, exactly as replay will read it from the record. So the first run and replay will see identical values, not just similar ones.

So how does place-order get its Date back? Reserve-stock will return the expiry as an ISO string, and pass a schema to step dot run. The schema will turn the string into a Date. Step dot run will apply the schema to the decoded value on the first run and on every replay, so both will see a Date. If the schema rejects the saved value, both runs will throw a schema error (WorkflowValueError: STEP_SCHEMA_INVALID).

Errors will get their own encoding. Only a value in an error position, such as a thrown step error or a cause, will come back as an error, so the crash report will stay a plain object. An error will keep its name, message, code, cause, and any durable fields. And once the shop registers the courier error class, replay will rebuild a real courier error, without calling its constructor, so the catch block will book the backup again.

This is a breaking change. After the upgrade, a step that returns a Date, a Map or a class instance will throw, and that includes executions already in flight when they reach such a step. So operators should fix those workflows, or let them finish, before upgrading. Steps recorded before the upgrade will keep their old decoder, so their replay will not change.

Before, Yieldstar saved values as JSON and quietly changed them on the way back, so replay could see a string where the first run saw a Date. With RFC 011, one codec will accept only values it can return unchanged, reject the rest on the first run, and give replay exactly what the first run saw.

Problem

serialise.ts stores each step response with JSON.stringify, using a replacer that converts Error instances with serializeError. On read, a reviver turns any error-like plain object, one with name, message and stack properties, back into an Error.

const event = yield* step.run("fetch-event", async () => ({
  name: "launch",
  message: "Ship it",
  stack: "web",
  at: new Date(),
}))

// First run: a plain object whose `at` is a Date.
// Replay: an Error, because the object has `name`, `message` and `stack`,
// and `at` is a string.

The serializer changes values in several ways:

• an error-shaped plain object becomes an Error;
• a Date becomes a string, and a Map or Set becomes {};
• undefined properties disappear, and a BigInt throws;
• a class instance loses its prototype; and
• a revived error loses its class, and its custom fields depend on serialize-error.

Two value models now coexist. RFC 003 and RFC 005 use DurableValue and its canonical encoding. RFC 001 stores params and context under "the runtime's event serialization", and RFC 008 compares idempotent params that way. Several RFCs defer error revival to this one.

Proposed solution

Every durable value will pass through one codec. The codec will accept DurableValue from RFC 003 and will reject anything else when it is produced.

// A Date is not a DurableValue, so this step throws INVALID_STEP_VALUE
// on the first run, and replay throws the same error.
yield* step.run("now", async () => new Date())

// A schema can turn a stored ISO string into a Date.
// The schema runs on the first run and on replay, so both see a Date.
const at = yield* step.run(
  "now",
  async () => new Date().toISOString(),
  { schema: z.string().pipe(z.coerce.date()) }
)

undefined and errors will be tagged durable values. They may appear anywhere a value can, and only a tagged value will decode as undefined or an error. Error-shaped plain objects will stay plain objects.

Proposed API

// RFC 003's DurableValue gains two tagged members.
type DurableValue =
  | null
  | boolean
  | number
  | string
  | undefined
  | Error
  | DurableValue[]
  | { [key: string]: DurableValue }

// Compile-time check for every recorded value. Unlike an index signature,
// it accepts interfaces.
type Durable<T> =
  T extends string | number | boolean | null | undefined | Error ? T
  : T extends readonly (infer U)[] ? readonly Durable<U>[]
  : T extends Function | Date | Map<any, any> | Set<any> | bigint ? never
  : { [K in keyof T]: Durable<T[K]> }

type DurableDecoding =
  | { status: "decoded"; value: DurableValue }
  | { status: "invalid"; message: string }

interface DurableValueCodec {
  encode(value: unknown): DurableValueEncoding
  decode(canonical: string): DurableDecoding
}

type WorkflowValueErrorCode =
  | "INVALID_STEP_VALUE"
  | "STEP_SCHEMA_INVALID"

declare class WorkflowValueError extends Error {
  readonly code: WorkflowValueErrorCode
  readonly stepKey: string
}

declare function registerErrorClass(
  name: string,
  errorClass: abstract new (...args: any[]) => Error
): void

interface StepRunner {
  run<T extends Durable<T>>(
    stepKey: string,
    fn: () => Promise<T>
  ): AsyncGenerator<StepResponse, T>

  run<S extends StandardSchemaV1>(
    stepKey: string,
    fn: () => Promise<StandardSchemaV1.InferInput<S>>,
    options: { schema: S }
  ): AsyncGenerator<StepResponse, StandardSchemaV1.InferOutput<S>>
}

durableValueCodec from RFC 003 will gain decode. DurableValue will gain undefined and errors; every other RFC 003 rule, and DurableValueEncoding, will stay as defined there.

Durable<T> will constrain every API that records a value: step results, store state, params, effect input and results, and workflow results. The runtime codec will still validate every value.

Required Semantics

One codec

The codec will encode every durable value:

• step.run results, and workflow results;
• select, when and take results, and store state;
• join results (RFC 009);
• admission params and start params (RFC 001 and RFC 003);
• effect inputs and results (RFC 005); and
• context.

Further rules:

• context will be encoded as entries with string keys and DurableValue values. It will decode back into a ReadonlyMap.
• undefined will be allowed anywhere: as a whole value, as an object property or as an array item. It will be encoded with a tag, so { a: undefined } and {} stay distinct.
• Tagged values will be objects with a $type key, such as { $type: "undefined" }.
• A plain object with its own $type key will be stored wrapped as { $type: "object", value }, so every plain object reads back exactly as written. Plain objects without a $type key will be stored unchanged.
• Workflow code will always receive the decoded value, on the first run and on replay. A first run will not return the original object.

Invalid values

• A step.run result that is not a DurableValue will be recorded and thrown as WorkflowValueError with INVALID_STEP_VALUE. Workflow code may catch it, and replay will rethrow it.
• An invalid workflow result will be recorded as a failed outcome with INVALID_STEP_VALUE. Its stepKey will be $$workflow-result$$.
• An invalid store state will make its store step throw INVALID_STEP_VALUE and will leave the store unchanged.
• Invalid admission params or context will return RFC 001's INVALID_EXECUTION_EVENT.
• An invalid value is not an infrastructure error. Treating it as one would redeliver the execution forever and run the function's side effects each time.

Schema decoding

• With schema, step.run will encode the function's result, decode it, validate the decoded value with the schema, and return the schema's output. It will do this on the first run and on replay.
• Standard Schema v1 has no encode step. The function's result must already be a DurableValue.
• The step record will note whether the schema accepted the value on the first run.
• If validation fails on the first run, the raw value will still be recorded, and both runs will throw WorkflowValueError with STEP_SCHEMA_INVALID.
• A schema that rejects on replay a value it accepted on the first run is a code change, not a step error. RFC 012 defines what happens.

Errors

• An error will be a tagged durable value. It may appear anywhere a value can: thrown, in a failed outcome, as cause, in an array such as WorkflowRaceError.errors, or inside an object that workflow code returns.
• Only a tagged value will be revived as an error. An error-shaped plain object will stay a plain object.
• The encoding will keep name, message, stack, code when it is a string, cause recursively, and own enumerable properties whose values are DurableValue. Other properties will be omitted.
• A thrown value that is not an error will be kept as thrown if it is a DurableValue. Otherwise it will be recorded as Error(String(value)).
• Revival will restore the class registered under the error's name. It will set the prototype and fields without calling the constructor.
• An error whose name is not registered will be revived as an Error with the same name, message, code, cause and fields.
• Registration is per process. A worker that has not registered a class will revive that class as a plain Error.
• Workflow code will always catch the revived error, on the first run and on replay.
• Yieldstar will register its own classes by default: RetryableError, WorkflowStartError, WorkflowEffectError, WorkflowJoinError, WorkflowCancelledError, WorkflowRaceError and WorkflowValueError.
• The standard JavaScript errors (TypeError, RangeError, SyntaxError, ReferenceError, EvalError, URIError and AggregateError) will also be registered by default.

Comparison

• Parameters will be compared by canonical encoding everywhere: start params (RFC 003), idempotent trigger params (RFC 008) and effect input (RFC 005).
• The canonical encoding will sort object keys and encode -0 as 0, as in RFC 003.
• context will not participate in any comparison.

This is a breaking change. A workflow that returns a Date, Map, Set, BigInt or class instance from a step, a store update or its result will throw INVALID_STEP_VALUE. The fix is to return a DurableValue and use a schema to transform it on read. There will be no compatibility mode, and values written by the old serializer will not be read.

Relationship to RFC 001

• params, context and result will use this codec. RFC 001's "runtime's event serialization" will mean this codec.
• A failed outcome's error will keep its class, code and cause, beyond RFC 001's name and message guarantee.

Relationship to RFC 003, RFC 005 and RFC 008

• All three will compare parameters by canonical encoding. RFC 008's "equal under the runtime's event serialization" will mean canonical equality.
• RFC 005's effect results may be undefined or contain errors, like any durable value.

Relationship to RFC 009, RFC 010 and RFC 011

• WorkflowJoinError, WorkflowCancelledError and WorkflowRaceError will be registered by default, so instanceof holds on replay.
• WorkflowJoinError.cause will keep the target's error class when it is registered in the joiner's worker.
• WorkflowCancelledError will keep reason and source, as RFC 010 requires.

Runtime changes

None. Runtime clients store encoded values as opaque strings. The schema flag lives inside the encoded step record.

Conformance tests

Value tests:

Case	Observable outcome
A step returns an error-shaped plain object	Replay returns the same plain object, not an Error.
A step returns a Date, Map, Set, BigInt or class instance	The first run throws INVALID_STEP_VALUE; replay rethrows it; workflow code can catch it.
A step returns undefined, { a: undefined } or [1, undefined]	Both runs return the same value; { a: undefined } keeps its key.
A step returns a plain object with keys that look like a tag	Replay returns the same plain object.
A step returns a valid value	The first run returns a decoded copy, equal to the value replay returns.
A workflow returns a Date	The outcome is failed with INVALID_STEP_VALUE.
Admission with a Date in params or context	It returns INVALID_EXECUTION_EVENT.
Context round trip	Every delivery receives a ReadonlyMap with equal entries.

Schema tests:

Case	Observable outcome
A schema transforms an ISO string into a Date	The first run and replay both return a Date.
The schema rejects the value on the first run	The raw value is recorded with the schema marked as rejecting it; both runs throw STEP_SCHEMA_INVALID.
The schema accepts the value on the first run	The record marks the schema as accepting it.

Error tests:

Case	Observable outcome
A step throws a registered custom error	Replay rethrows the same class, with code, cause and fields.
A step throws an unregistered custom error	Replay rethrows an Error with the same name, message and code.
A registered class has a constructor with required arguments	Revival succeeds without calling it.
An error has a non-durable property	The property is absent on the first run's caught error and on replay.
A step throws an error with a stack	Replay rethrows it with the same stack.
A step throws a string or a plain object	Replay rethrows the same value.
A step throws a non-durable value	Both runs throw Error(String(value)).
Workflow code returns { ok: false, error }	The outcome's result keeps error as an error of the same class.
A race error escapes the workflow	The failed outcome keeps errors with each branch's error.
A step throws a TypeError	Replay rethrows a TypeError.
Each built-in Yieldstar error is thrown	Replay rethrows the same class; instanceof holds.
A failed execution is read through poll, wait or join	The error keeps its registered class, code and cause.

Comparison tests:

Case	Observable outcome
Params differ only in key order	They are equal for start, idempotent trigger and effect input.
Params differ only in -0 and 0	They are equal.
Requests differ only in context	They are equal.
Params differ only in { a: undefined } and {}	They are not equal.

Scope

This RFC will not specify:

• replay stability across code changes, call-site keys or versioning, which RFC 012 will cover; or
• encodings for values beyond DurableValue, such as Date or BigInt. A schema can transform a stored value into those types.

Not Specified

This RFC will not choose:

• the fields inside each tag, beyond $type; or
• how the schema flag is stored in a step record.