RFC 004: Durable External Effect

This is a design RFC for step.effect and an EffectClient in @yieldstar/core, and it depends on RFCs 001, 002, 003 and 011 linked in the body. Read those linked RFCs and the existing step.run behavior before forming any view. Done means the design is discussed and accepted by maintainers, with the conformance table in the body turned into tests, not a code change.

Abstract

A worker can crash after an external system finishes an operation but before Yieldstar saves the result. Therefore, retrying can perform the operation twice, while stopping can leave the workflow without its result.

We will solve this with step.effect, which will use a caller-supplied idempotency key to ask the external service what happened after an uncertain attempt. If the service reports that the operation completed, the step will return its result; if the service proves that it did not start, the step can safely try again.

https://github.com/user-attachments/assets/9cc84c5b-2cf9-49e1-958c-707dcf6f33ca

Video transcript

Back at the shop. Order 42 runs as an execution of charge-order, and its first step, called charge, charges the customer's card. The charge step is an ordinary step, made with step dot run. Inside it, the shop calls the payment provider. The provider takes the money and returns a payment. When the function returns, Yieldstar saves the payment as the charge step's result.

Remember the rule from RFC 003: a step with a saved result is skipped, and a step without one runs again. Now say the provider takes the money, but the worker crashes before Yieldstar saves the payment. Or the provider's reply is lost on the network. Either way, the charge step has no saved result. So when Yieldstar runs charge-order again, by the rule, the charge step runs again, and the provider takes the money a second time. The customer pays twice for one order.

So why not just stop? Say Yieldstar refuses to run the charge step again after a crash. Then the customer pays once, but charge-order never learns about the payment, so it never ships the order. Yieldstar is stuck between two bad choices. If it retries, it may charge twice. If it stops, it may lose a payment. And from inside the worker, Yieldstar cannot tell which case it is in. The crash could have come before the provider took the money, or after. This is the heart of the problem: only the payment provider knows what happened.

RFC 003 closed a gap like this one for starting workflows. There, Yieldstar saved the new execution and its request ID together, all-or-nothing, because both lived in Yieldstar's own store. A card payment does not live in Yieldstar's store. It lives at the payment provider. No single write can save a payment at the provider and a step result in Yieldstar together. So Yieldstar cannot close this gap. Instead, after a crash, it will have to ask the provider what happened inside the gap.

RFC 004 will add a step for calls to outside systems, called step dot effect. The shop will give it a step key, charge, and a description of the call. The description will hold an effect ID, which names the payment itself. It will hold the input: the amount. It will hold a perform function, which charges the card under that effect ID. And it will hold a lookup function, which asks the provider what it knows about that effect ID.

Why does the charge step need a second name? The step key, charge, names a place in one execution of charge-order. The effect ID names one payment at the provider. Those are different things. Say the shop's support tool also needs to check on this payment. The support tool has no charge-order execution, but it can use the same effect ID. So every caller that means this payment will use the same effect ID, and no caller will reuse it for a different payment. The shop will build the effect ID from everything that picks out one payment: the shop's account, the kind of operation, and the order.

This is how step dot effect will know that a call may have happened. Before it calls perform, Yieldstar will write an effect record for the effect ID, marked pending. This order is important: the record will always exist before the money moves. Now replay the crash from earlier. The provider took the money, and the worker crashed before Yieldstar saved the step. Charge-order runs again, and by the rule, the charge step runs again. But this time, Yieldstar will find the pending record. Pending means a call may have started, and may have finished. So Yieldstar will not call perform. It will call lookup.

Lookup will give one of three answers. Settled: the provider has the payment, and here it is. Yieldstar will take that payment as the result, and the customer pays once. Not found: the provider has no payment under that effect ID. Then Yieldstar will call perform. Unknown: the provider can't say yet. Then Yieldstar won't guess. It will ask to be woken after a delay, say one minute, pause the step, and look up again when it wakes. A lookup or perform call that fails will count as unknown too.

Not found carries the weight of the whole design, so it needs a strict meaning. It is not enough that the provider has no payment at this moment. Say the first charge is slow, and still in flight inside the provider. If lookup said not found, Yieldstar would charge again, and then the slow charge would land too. Two payments. So not found must prove that no earlier call exists, is running, or can still finish. The provider must keep that promise, and two more: repeated perform calls with one effect ID make one payment, and a settled answer never changes. If the provider can't keep these promises, step dot effect can't make the charge safe.

Once perform or lookup returns a payment, Yieldstar will settle the effect record. That means it will store the payment in the record, and the record will never change again. Then Yieldstar will save the charge step's result, as before. If the worker crashes between those two saves, the replay will find a settled record and return its payment without calling the provider at all. And if another execution was waiting on the same effect, Yieldstar will wake it, and it will read the same payment.

One question remains, and RFC 003 asked it too: what if the replay asks for something different? Say that while the charge for order 42 was pending, the shop deployed new code that charges 59.99 instead of 49.99, under the same effect ID. Yieldstar will compare the operation name and the input with the effect record, using the same fixed form of JSON as RFC 003. When they differ, Yieldstar will refuse with a mismatch error (EFFECT_REQUEST_MISMATCH), and the workflow can catch it. Yieldstar won't return a payment for the wrong amount, and it won't charge the card again.

Before, a step that calls an outside system has a gap. If the worker crashes after the provider acts but before Yieldstar saves the result, Yieldstar must choose between charging twice and losing the payment. With RFC 004, Yieldstar will write a pending record before every call, and after a crash, it will ask the provider instead of guessing. However many times the charge step runs, the customer will pay once.

Problem

step.run records its result after its function returns.

const resource = yield* step.run(`provision:${accountId}`, async () => {
  const resource = await cloud.provision(configuration)

  // The resource now exists, but Yieldstar has not recorded the result.
  // If the worker crashes here, replay will provision it again.
  return resource
})

If the cloud service creates the resource but the response is lost, retrying this step calls provision again.

Proposed solution

step.effect will give the external operation a stable identity. After an uncertain attempt, it will look up that identity before deciding whether it is safe to perform the operation again.

const resource = yield* step.effect(`provision:${accountId}`, {
  // This ID belongs to the external operation, not this workflow execution.
  id: `account:${accountId}`,
  input: configuration,
  lookup: id => cloud.lookup(id),

  // Yieldstar will call perform again only after lookup proves that no
  // earlier call exists or can still finish.
  perform: (id, configuration) => cloud.provision(id, configuration),
  retryIn: 60_000,
})

Proposed API

step.effect will separate the workflow step from the external operation. The step key will identify where the call occurs in one workflow execution; the effect ID will identify the operation in the external service.

input, lookup results, and performed results will be DurableValue data. RFC 003 defines DurableValue, its validation rules, and its canonical encoding. durableValueCodec will be the only validator and canonical encoder for these values.

type EffectLookup<T extends DurableValue> =
  | { status: "settled"; result: T }
  | { status: "not-found" }
  | { status: "unknown" }

type EffectDescriptor<T extends DurableValue> = {
  effectId: string
  operation: string
  input: { [key: string]: DurableValue }
  retryIn: number
  lookup(effectId: string): Promise<EffectLookup<T>>
  perform(effectId: string): Promise<T>
}

interface StepRunner {
  effect<T extends DurableValue>(
    stepKey: string,
    effect: EffectDescriptor<T>
  ): AsyncGenerator<StepResponse, T, StepResult | StepError>
}

stepKey will identify the step within one workflow execution. effectId will identify the external operation, even when several runtimes can call the same external system.

Every caller of one external operation must use the same effectId. No caller may use that ID for a different operation.

operation will name the version of the behavior defined by input, lookup, and perform. It will not change after the effect record is created.

retryIn must be finite and non-negative. It will set how long the step waits before it looks up an uncertain outcome again.

step.effect will not add authorization to an effect result. A workflow that submits the same effectId, operation, and input will receive the stored result.

An application should build effectId from every part needed to identify one external operation. Depending on the application, those parts may include:

• the provider account;
• the tenant;
• the kind of operation; and
• the business identity.

External system contract

Every runtime that calls the external system will need to use the same effectId for the same operation. The external system must provide these guarantees for that identity:

1. Concurrent or repeated perform calls refer to one operation and return one result.
2. not-found proves that no accepted, running, or completed call exists and that no earlier call can later finish.
3. settled returns the same result as every successful perform call and remains stable.
4. unknown means neither perform nor absence is safe to assume. A later lookup may try again.

A rejected lookup or perform call will also be an unknown outcome. If the external system cannot provide these guarantees, step.effect will not make the operation safe.

Runtime API

step.effect will run under the delivery acknowledgement from RFC 003, the execution record from RFC 001 and the leases from RFC 002.

This RFC will add an EffectClient to @yieldstar/core:

interface EffectClient {
  begin(request: {
    operation: string
    effectId: string
    input: { [key: string]: DurableValue }
    executionId: string
    stepKey: string
  }): Promise<
    | { status: "new" }
    | { status: "pending" }
    | { status: "settled"; result: DurableValue }
    | { status: "rejected"; code: "EFFECT_REQUEST_MISMATCH" }
  >

  settle(request: {
    operation: string
    effectId: string
    result: DurableValue
  }): Promise<
    | { status: "settled"; result: DurableValue }
    | {
        status: "conflict"
        result: DurableValue
        conflictingResult: DurableValue
      }
  >

  wait(request: {
    effectId: string
    executionId: string
    stepKey: string
    wakeAt: number
  }): Promise<{ wakeAt: number }>
}

class EffectClientInvariantError extends Error {
  code: "EFFECT_RUNTIME_INVARIANT"
}

interface WorkflowEffectError extends Error {
  code: "INVALID_EFFECT_REQUEST" | "EFFECT_REQUEST_MISMATCH"
}

Required Semantics

• Each effect will be identified by effectId. Its operation and input will be immutable, and a settled result will never change or be deleted.
• begin will create a pending effect before returning new. If the effect already exists, a different operation or input will return EFFECT_REQUEST_MISMATCH; matching values will return the effect's current state. Values will match when their canonical encodings match.
• settle will settle a pending effect only once. Concurrent calls with the same encoded result will return that result. A call with a different result will keep the first result, return conflict with both values, and preserve a permanent conflict report that runtime operators can inspect.
• wait will register the calling execution as a waiter and request a wake for it at wakeAt through RFC 001. It will carry no workflow event. Registration and the wake request will be all-or-nothing.
• Repeating wait for the same effect, execution, and step will return the first wakeAt without adding another registration.
• When settle succeeds, every waiting execution will be woken. No execution will remain registered without a pending or delivered wake.
• Wakes will be pending wakes on the execution record (RFC 001). They will survive a runtime restart.
• A repeated wake request after a lost response will merge into the execution's pending wake. Any extra delivery will be safe because it reads the same effect and step records.
• Errors will use the tagged error encoding from RFC 011. A plain object that resembles an error will remain a plain object.
• begin and settle will be idempotent by effect ID and will not be fenced. wait will be unfenced. The step records that step.effect writes will be fenced heap writes (RFC 002).
• Cancelling an execution with a pending effect will not cancel the external operation. The effect record will stay pending and may still settle (RFC 009).
• Effect records will not be pruned with an execution (RFC 013).

Driver behavior

step.effect will first ask whether a step result is already recorded. When one exists, it will return or throw that result without calling the client or a callback. Otherwise:

Outcome	Behavior
Invalid effectId, operation, input, or retryIn	The driver will record and throw INVALID_EFFECT_REQUEST before begin.
begin returns new	The driver will call perform.
begin returns pending	The driver will call lookup.
begin returns settled	The driver will record and return its result without calling a callback.
begin returns EFFECT_REQUEST_MISMATCH	The driver will record and throw that code without calling a callback.
perform returns or lookup returns settled	The driver will validate the result, call settle, then record and return its result.
lookup returns not-found	The driver will call perform.
A callback rejects or lookup returns unknown	The driver will call wait, then suspend the step.
settle reports a conflict	The driver will report it and record the first result.
begin, settle, or wait throws an ordinary error	The driver will record no step and request another delivery with EFFECT_CLIENT_UNAVAILABLE.
A callback returns a malformed response or invalid value	The driver will record no step, retain the pending effect, and request another delivery with EFFECT_HANDLER_INVALID.
EffectClient throws EffectClientInvariantError	The driver will record no step and request another delivery with EFFECT_RUNTIME_INVARIANT.

A suspended effect step will remain unfinished. A later delivery will start the effect step again from begin. The worker will acknowledge suspension only after both wait and the unfinished step record succeed.

INVALID_EFFECT_REQUEST and EFFECT_REQUEST_MISMATCH will be recorded as WorkflowEffectError values. Workflow code may catch them, and their code will survive serialization.

Three error codes will request another delivery instead of reaching workflow code:

• EFFECT_CLIENT_UNAVAILABLE will describe an uncertain failure that may recover without intervention.
• EFFECT_HANDLER_INVALID will report an invalid callback response or result.
• EFFECT_RUNTIME_INVARIANT will report invalid data or behavior inside the runtime.

The last two errors will require operator repair if they persist. All three will leave the delivery unacknowledged under RFC 003. RFC 002 will release its lease and apply the redelivery delay.

If workflow code changes while an effect is pending, the new code must still call step.effect with the same effectId. Its callbacks must keep the same external-system guarantees. Otherwise, Yieldstar may be unable to discover or safely repeat the operation.

Conformance tests

Runtime-adapter tests will exercise EffectClient and wake delivery:

Operation or race	Observable outcome
begin creates or reads an effect	One effect exists for the ID; repeated matching calls return its current state.
begin receives changed operation or input	It rejects without changing the effect.
settle receives equal results concurrently	One settled result exists.
settle receives different results	The first result remains settled and one conflict report contains both values.
Failure before wait commits	No wake is scheduled; a repeated call succeeds as a first call.
wait commits and its response is lost	Repeating it returns the first wake time and adds nothing.
Settlement races with wait	The waiting execution is woken; no registration is left without a wake.
The runtime restarts with stored wakes	Delivery of those wakes resumes.
A wake request commits but its response is lost	Resubmission merges into the same pending wake.

Workflow-driver tests will use a fake client and callbacks:

Path or interruption	Observable outcome
New effect	begin, perform, and settle run in order.
Pending effect with not-found	begin, lookup, perform, and settle run in order.
Already-settled effect	No callback runs and the recorded result is returned.
During perform, after external completion, or after its response	lookup runs before another perform.
After lookup returns not-found	lookup runs again before perform.
After lookup returns settled but before settle	lookup runs again and the same result is stored.
After settle but before the step result	The effect record supplies the result without an external call.
After the step result but before workflow continuation	The recorded step supplies the result.
lookup returns unknown or a callback rejects	wait completes before the step suspends.
Invalid request	INVALID_EFFECT_REQUEST is recorded and reaches workflow code without calling the client.
Changed operation or input	EFFECT_REQUEST_MISMATCH is recorded and reaches workflow code without a callback.
Client availability failure	No step is recorded and delivery retry carries EFFECT_CLIENT_UNAVAILABLE.
Malformed lookup or invalid callback result	No step is recorded, the effect stays pending, and delivery retry carries EFFECT_HANDLER_INVALID.
Client invariant	No step is recorded and delivery retry carries EFFECT_RUNTIME_INVARIANT.
Conflicting settlement	The conflict is reported and the first result is recorded and returned.

Serialization tests will round-trip every valid durable value, including an error-shaped object, without changing its type or value. They will reject every invalid value. A suspended effect step will remain unfinished after serialization and deserialization.

Integration tests will cover EffectClient through delivery acknowledgement:

Interruption	Observable outcome
wait commits, then the worker fails before the unfinished step record	The lease is released or expires; the next delivery reuses the first registration and wake.
The unfinished step record commits, then the worker fails before replying	The lease is released or expires; the next delivery remains safe.
A requested wake becomes due	The execution is delivered once, and acknowledgement clears the wake.
settle commits with waiting executions	Each is woken and reads the settled result.
EffectClient fails inside the worker	The worker replies retry, deliver rejects, the lease is released, and no step result exists.

Not Specified

This RFC will not choose:

• how effect records and waiter registrations are stored;
• the driver-internal step responses that carry an effect request or a suspension; or
• when waiter cleanup runs inside settlement.