RFC 0009: Recoverable creates

Start by reading RFC 0009 and the referenced RFCs 0001, 0002, 0006, 0010, and Yieldstar 005 to understand the proposed recovery and read contracts. The RFC points to @notation/resource, provider create and read implementations, readResourceOperation, state persistence, planning, dashboard, and orphan sweep; trace those entry points and identify the affected resources, including the four that lack reads. Done means the engine records creates before calling providers, recovers typed already-exists errors, and compares config-shaped read params without merging desired values.

Abstract

0001 replays a provider call when a crash falls between the provider's acknowledgement and the checkpoint, so it requires creates to be idempotent. No provider meets that. A replayed CreateRole fails with EntityAlreadyExists, and nothing handles it. Even if the engine read the existing role, it couldn't decide what to do with it: read returns the API's response shape, merged with desired params, so a found resource can't be compared with its config.

We will solve this by recording a create before it is attempted, and by giving providers a typed "already exists" error and a read that returns params in config shape. After an uncertain create, the engine will find its own record, read the resource, reclaim it, and diff it like any other. This provides what 0001's idempotent-create assumption needs. 0001 is not amended.

https://github.com/user-attachments/assets/a176a36a-7a38-4dfd-b852-43331a6964f0

Video transcript

When you run notation deploy, Notation's reconciler creates each resource in two steps. Take, for example, the IAM role of a Lambda function called orders. First, the reconciler asks IAM to create the role, orders-role. Then the reconciler writes a state record: Notation's record of what exists. Since RFC 1, the reconciler also writes the result of each step to a log outside the process. So after a crash, you can rerun the same execution, and the reconciler continues where it stopped.

But there is a gap. Say IAM creates the role, and then the process crashes, before the reconciler can write IAM's answer to the log. You rerun the execution. The log has no result for the create step, so the reconciler asks IAM to create the role again. RFC 1 assumes that this is safe: that a provider can repeat a create. This is important, because no provider can. IAM refuses the second create with an error that says the role already exists (EntityAlreadyExists). Notation has no code that handles that error, so the deploy fails.

Now say you give up on that execution, and start a new run. The new run has its own log, so it cannot read the old log. And state has no record of the role, because the reconciler writes the state record only after the create returns. So the new run has no evidence that the reconciler ever tried. The new run asks IAM to create the role, and IAM refuses again.

RFC 10 will make this worse. RFC 10 will give each role a generated name that ends in a random part (orders-app-prod-orders-role-7f3a9c21). Until the create returns, that name will exist only in the old log. A new run cannot make the same name again, so it will create a second role. The first role will stay in AWS, with nothing to manage it. And a resource that has no tags, such as an EventBridge rule, will have no record and no tags. So the new run will treat that rule as someone else's resource, and stop.

Suppose the reconciler tried to recover from "already exists". Problem 1: the reconciler cannot recognise the error. Each provider throws the error of its own AWS SDK. IAM throws one error, and CloudWatch Logs throws another. To the reconciler, "already exists" looks the same as any other failure.

Problem 2: if the reconciler found the existing role, it could not compare the role with your config. To compare, the reconciler reads the role from IAM. But a read returns the shape of the AWS API, not the shape of your config. For example, IAM returns the trust policy of the role URL-encoded, and the permissions boundary as an object. Both fields are immutable, so a difference in either field forces a replacement. Until a recent fix to the role's read, drift detection saw a difference on every deploy, and replaced the role of every Lambda on every deploy. The role's read is now fixed, but the reads of the other resources still return API shapes.

Problem 3: the reconciler merges the read with your config. It starts with the params from your config, and writes the fields of the read over them. The result is a good output for the resources that depend on the role. But drift detection and the ownership check also use the result, as if it were what AWS holds. Say the read does not return a field. After the merge, that field has the value from your config. So a field that the read did not return looks the same as a field that matches.

Now, the fix, in three parts. Part 1: before the reconciler calls create, it will write a state record with the status "creating". The record will hold the key of the role, including any generated name. The record goes in state, not in the log, because state is what a new run reads. So a new run will find the record, and will know that the reconciler tried, and what name it used. Under RFC 10, a resource with a state record and no tags will be presumed to be ours.

A resumed execution will also check before it repeats a create. The create step will become a durable effect from Yieldstar, with two functions: lookup, which is the provider's read, and perform, which is the provider's create. After an attempt with no recorded result, the step will first call lookup. The step will call create again only if lookup does not find the role.

Part 2: each provider will turn its own "already exists" error into one typed error from Notation: resource already exists error (ResourceAlreadyExistsError). IAM's error will become this error. When create throws it, and the role has a state record, the reconciler will read the role, reclaim it, and then diff it like any other resource. The diff decides: do nothing, update the role, or replace it.

Part 3: a new contract for read. Read will return two parts. The first part, params, will be in the shape of your config. The second part, output, will hold values that only AWS knows, such as the ARN of the role. For the role, the provider decodes the trust policy, and keeps only the ARN of the permissions boundary. Only the provider knows how to turn the API shape back into config shape, so this work belongs in the provider. And every resource must implement read. Four resources have no read today, such as the policy attachment and the log group. Each of them will gain one.

The reconciler will never merge the params of the read with your config. Drift detection, the ownership check and reclaim will compare your config with the params exactly as read. The resources that depend on the role will get your params plus the output of the read. And a field that read does not return will be unverified. The plan will list it, drift detection will ignore it, and it will never force a replacement. For a nested field, drift detection will compare only the keys that your config sets. For example, if your config sets only the log format of the orders Lambda, the log group that AWS fills in is unverified.

This matters most for replacement. To replace a role that the reconciler found, the reconciler deletes a real role. So only evidence will trigger a replacement: an immutable or key field that the read returned, that is valid, and that differs from your config. The reconciler will check each read against the value types of RFC 6. If the read is malformed, the reconciler will leave the role as it is, and the plan will show "blocked: invalid read". This follows the rule from RFC 2: the plan may be more careful than the deploy, but never less careful.

Two limits remain. Limit 1: for some resources, AWS assigns the key. An API Gateway API is one of them. AWS gives each new API a new ID, so a second create never fails with "already exists". Instead, it makes a second API. To fix this, providers need idempotency tokens, and this is out of scope. Limit 2: if the AWS API never returns a field, that field stays unverified, so drift in that field goes unnoticed.

Before, if the process crashed during a create, the reconciler sent the create again, IAM refused it, and the deploy failed. With this RFC, the reconciler will record each create before it calls the provider, will recognise "already exists", and will read the role in the shape of your config. Then an ordinary diff will decide whether to keep the role, update it, or replace it.

Problem

• A create can't recognise its own earlier attempt. The state record is written after create returns. A crash before that leaves no evidence that the engine tried. A tagged resource carries evidence on the cloud object (0010), but an untagged one, such as an EventBridge rule, has none. Under 0010 a new run finds it with no record and no tags, and blocks it as foreign.
• Journals don't outlive executions. Resuming the same execution can consult its journal. A new run after clearDeploymentHold can't, because the abandoned execution's journal is gone.
• "Already exists" is untyped. Each provider throws its own SDK error, so the engine can't tell it from a real failure.
• A read can't be compared with config. IAM returns AssumeRolePolicyDocument URL-encoded and PermissionsBoundary as { PermissionsBoundaryArn, PermissionsBoundaryType }. Both are immutable, so until main's f150ec6 fixed the role's read, drift detection replaced every Lambda's role on every deploy. The other providers' reads still return API shapes.
• A read result does two jobs. readResourceOperation returns { ...params.resourceParams, ...remote }. That is a valid output for dependents, but drift and ownership also use it as an observation of the cloud. A field the read didn't return silently takes the desired value, so "not exposed" looks the same as "matches".

Proposed solution

The "creating" record

Before calling create, the engine writes a state record saying it is about to create the resource. The record holds the key and any generated name.

// Step state:persist:orders-role:creating, run before CreateRole is called.
{ status: "creating", key: { RoleName: "orders-app-prod-orders-role-7f3a9c21" } }

• A new run after an abandoned execution finds the record. It is a state record, so 0010's rule applies: with a state record, unknown is presumed ours.
• A resumed execution is covered by yieldstar 005. The create becomes step.effect(…, { lookup: provider.read, perform: provider.create }), so after an uncertain attempt the step looks the resource up before it calls create again.
• The generated name survives a crash. This removes the crash case from 0010's limit that a lost resource with a generated name can't be found by its key.
• Plan, the dashboard and the orphan sweep understand the record. An orphaned "creating" record may point at nothing, so deleting it accepts not-found.

ResourceAlreadyExistsError

@notation/resource gains ResourceAlreadyExistsError, next to ResourceNotFoundError. Each provider maps its own error to it, e.g. IAM's EntityAlreadyExists. When create throws it and a record exists, the engine reads the resource, reclaims it, then diffs to choose noop, update or replace.

The read contract

read returns { params, output }, with params in config shape. The role's read on main today returns the API's Role, with only the two immutable fields reshaped:

// before: API shape plus two fixed fields; the engine then merges desired params over it
read: async (key) => toReadResult((await iamClient.send(new sdk.GetRoleCommand(key))).Role!)

// after
read: async (key) => {
  const { Role } = await iamClient.send(new sdk.GetRoleCommand(key))
  return {
    params: {
      RoleName: Role.RoleName,
      Path: Role.Path,
      AssumeRolePolicyDocument: decodePolicyDocument(Role.AssumeRolePolicyDocument),
      PermissionsBoundary: Role.PermissionsBoundary?.PermissionsBoundaryArn,
      Description: Role.Description,
      MaxSessionDuration: Role.MaxSessionDuration,
    },
    output: { Arn: Role.Arn, RoleId: Role.RoleId },
  }
}

• Never merged. The observation is the read's params as returned. The output for dependents is the desired params plus the read's output. This replaces the { ...desired, ...remote } merge in readResourceOperation. Ownership, drift and adoption all use the unmerged read.
• Unverified fields. A field read doesn't return is unverified. Plan lists it, drift ignores it, and it never forces a replacement.
• Nested keys. Drift compares only the nested keys the config sets. Keys the cloud fills in are unverified, e.g. Lambda's LoggingConfig.LogGroup and log levels when config sets only LogFormat.
• Validation. read results are validated by the fields' value types (0006). A malformed result raises ReadValidationError: the resource is left as it is, plan shows "blocked: invalid read", and it is never replaced.
• Replacement needs evidence. Only a present, valid immutable or key field that differs can trigger a replacement.

Every resource must implement read, since recovering from "already exists" needs one. The providers that have a read change it: LambdaIamRole, LambdaFunction, API Gateway Api, Stage, Route, Authorizer and LambdaIntegration, the EventBridge rule, and std/fs Zip and File. The role's and the function's reads already return config-shaped fields since f150ec6; they still split into params and output. Four resources have no read and gain one: LambdaRolePolicyAttachment, LambdaLogGroup, LambdaApiGatewayV2Permission and LambdaEventBridgeRulePermission. The log group's is needed for its recovery in 0010.

State writes

Each state write has its own key. The final write keeps state:persist:<id>, and extra writes are suffixed: state:persist:<id>:creating, :retired (0010) and :settle:<n> (0011). With yieldstar 006, opening a store handle is no longer a step, and the "creating" write is a getOrCreate with its own key. The workarounds for opening a store once per execution go: main's state:create bypass, 0010's separate dropRetired step and 0011's handle reuse.

Rationale

Only the engine knows it tried. Tags can mark a resource but can't exist before it, and nothing marks an untagged one, so the evidence has to be written before the call. It belongs in state rather than the journal because state is what a new run reads. Providers signal "already exists" and the engine decides, so recovery is one rule in the engine rather than idempotency logic in every provider.

The decision after "already exists" is an ordinary diff, which needs the found resource's params. Only the provider knows how to turn the API's shape back into config shape. Treating missing and malformed fields as unverified, and stopping on an invalid read, follows 0002's invariant that plan may be more cautious than deploy, never less: for a found resource, replacing it deletes something, and stopping is the cautious choice.

Limits

• A resource whose key the cloud assigns, such as an API Gateway API, never reports "already exists", so a replayed create makes a duplicate. That needs provider idempotency tokens, such as ClientToken, and is out of scope.
• A field a provider's API doesn't expose can never be verified, so drift in it goes unnoticed.