`transition_state` after `get_and_lock_for_update` checks the caller's copy, not the locked row

Open
#154 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
elixir, postgresql
Domain
backend, databases

Research direction

Start with lib/ash_state_machine.ex:142-147, where TransitionState.change/3 reads the old state, and trace how get_and_lock_for_update reloads the record before the update. Reproduce with locked_transition.exs against local PostgreSQL, then add regression coverage for stale copies and concurrent transitions. Done means stale transitions return NoMatchingTransition and only one concurrent confirmation succeeds.

Written by the indexing model from the issue text.

Description

Code of Conduct
  • I agree to follow this project's Code of Conduct
AI Policy
  • I agree to follow this project's AI Policy, or I agree that AI was not used while creating this issue.
Versions

ash_state_machine 0.2.13 (also main b22e138), ash 3.37.0, ash_postgres 2.15.1
Elixir 1.20.0, Erlang/OTP 28, PostgreSQL 17

Operating system

macOS 26

Current Behavior

Description drafted with an AI assistant; repro run and verified by me.

In an action that locks the record and then transitions it, transition_state checks the transition against the copy the caller passed in, not the locked row. So a stale copy can make a transition the stored state doesn't allow.

For example, cancelling an order locks it first, so that it can't be cancelled after it's been paid:

update :cancel do
  require_atomic? false
  change get_and_lock_for_update()
  change transition_state(:cancelled)
end

If the caller's copy is still :open but the order has since been paid, I'd expect NoMatchingTransition, with the order staying paid. Instead the paid order is cancelled. And if eight callers confirm the same order at once from the open copy they each read, all eight succeed and send a receipt:

cancel on a stale open copy of a paid order: {:ok, _}, stored state: :cancelled
cancel_atomic on a stale open copy of a paid order: AshStateMachine.Errors.NoMatchingTransition, stored state: :paid
cancel_always_atomic on a stale open copy of a paid order: AshStateMachine.Errors.NoMatchingTransition, stored state: :paid
8 concurrent confirm: 8 succeeded, 8 receipts sent

The atomic action without a lock (cancel_atomic) refuses the transition, because the check is in the UPDATE. So does the same locked action with transition_state(:cancelled), always_atomic?: true.

It looks like transition_state/2 reads the old state from changeset.data when the changeset is built (lib/ash_state_machine.ex:142-147), from TransitionState.change/3. get_and_lock_for_update re-reads and locks the row later, in a before_action, and nothing checks the transition again. The non-atomic UPDATE filters on the primary key only, so the lock just serialises a decision that's already been made. Declaration order doesn't change it, since changes always run before hooks.

This matters more since ash 3.35.0, where get_and_lock_for_update refuses to run atomically (ash-project/ash#3008). An action combining the lock with transition_state now needs require_atomic? false, which puts it on this path. Before that, the lock was silently skipped and the atomic check made it look right.

The getting started guide says whether a transition is allowed "is determined later by the state_machine.transitions configuration", and I read that as happening against the stored record. I haven't found anything about locking in the docs. erlef-cna/varsel works around it with a before_action that re-checks the state after the lock, so that "a stale in-memory struct cannot resurrect a resolved proposal". I wrote up the alternatives in more detail in this gist.

Reproduction

elixir locked_transition.exs. It needs a local Postgres (PGUSER/PGPASSWORD/PGHOST, default postgres/postgres/localhost) and creates the database ash_state_machine_locked_transition. ASH_STATE_MACHINE_REF=<sha> runs it against a commit.

locked_transition.exs
Mix.install(
  [
    case System.get_env("ASH_STATE_MACHINE_REF") do
      nil -> {:ash_state_machine, "~> 0.2.13"}
      ref -> {:ash_state_machine, github: "ash-project/ash_state_machine", ref: ref, override: true}
    end,
    {:ash_postgres, "~> 2.15"}
  ],
  consolidate_protocols: false,
  config: [ash: [default_string_length_count: :codepoints]]
)

Logger.configure(level: :warning)

Application.put_env(:repro, Repro.Repo,
  username: System.get_env("PGUSER", "postgres"),
  password: System.get_env("PGPASSWORD", "postgres"),
  hostname: System.get_env("PGHOST", "localhost"),
  database: "ash_state_machine_locked_transition",
  pool_size: 10,
  log: false
)

defmodule Repro.Repo do
  use AshPostgres.Repo, otp_app: :repro
  def installed_extensions, do: ["ash-functions"]
  def min_pg_version, do: %Version{major: 16, minor: 0, patch: 0}
end

defmodule Repro.Order do
  use Ash.Resource,
    domain: Repro.Domain,
    data_layer: AshPostgres.DataLayer,
    extensions: [AshStateMachine]

  postgres do
    table "orders"
    repo Repro.Repo
  end

  state_machine do
    initial_states [:open]
    default_initial_state :open

    transitions do
      transition :pay, from: :open, to: :paid
      transition :cancel, from: :open, to: :cancelled
      transition :cancel_atomic, from: :open, to: :cancelled
      transition :cancel_always_atomic, from: :open, to: :cancelled
      transition :confirm, from: :open, to: :paid
    end
  end

  attributes do
    uuid_primary_key :id
  end

  actions do
    defaults [:read, create: []]

    update :pay do
      change transition_state(:paid)
    end

    # Lock the order, then cancel it.
    update :cancel do
      require_atomic? false
      change get_and_lock_for_update()
      change transition_state(:cancelled)
    end

    # Lock the order, confirm the payment, and send a receipt.
    update :confirm do
      require_atomic? false
      change get_and_lock_for_update()
      change transition_state(:paid)
      change after_action(&Repro.Order.send_receipt/3)
    end

    # No lock, atomic: the transition is checked in the UPDATE.
    update :cancel_atomic do
      change transition_state(:cancelled)
    end

    # The lock, with the transition forced atomic.
    update :cancel_always_atomic do
      require_atomic? false
      change get_and_lock_for_update()
      change transition_state(:cancelled), always_atomic?: true
    end
  end

  def send_receipt(_changeset, order, _context) do
    Repro.Repo.query!("INSERT INTO receipts (order_id) VALUES ($1)", [Ecto.UUID.dump!(order.id)])
    {:ok, order}
  end
end

defmodule Repro.Domain do
  use Ash.Domain, validate_config_inclusion?: false

  resources do
    resource Repro.Order
  end
end

defmodule Repro do
  def setup! do
    Repro.Repo.__adapter__().storage_down(Repro.Repo.config())
    :ok = Repro.Repo.__adapter__().storage_up(Repro.Repo.config())
    {:ok, _} = Repro.Repo.start_link()

    # AshPostgres's SQL functions (ash_raise_error), for the atomic transition check.
    nil
    |> AshPostgres.MigrationGenerator.AshFunctions.install()
    |> then(&Regex.scan(~r/execute\(\"\"\"\n(.*?)\n\s*\"\"\"\)/s, &1, capture: :all_but_first))
    |> Enum.each(fn [sql] -> Repro.Repo.query!(sql) end)

    Repro.Repo.query!("CREATE TABLE orders (id uuid PRIMARY KEY, state text NOT NULL)")
    Repro.Repo.query!("CREATE TABLE receipts (order_id uuid NOT NULL REFERENCES orders(id))")
  end

  def update(order, action) do
    case order |> Ash.Changeset.for_update(action) |> Ash.update() do
      {:ok, _} -> "{:ok, _}"
      {:error, %{errors: [error | _]}} -> inspect(error.__struct__)
    end
  end

  def state(order), do: Ash.get!(Repro.Order, order.id).state

  # A copy of an open order, read before another process paid it.
  def stale_copy_of_paid_order do
    order = Ash.create!(Repro.Order, %{})
    order |> Ash.Changeset.for_update(:pay) |> Ash.update!()
    order
  end

  def run do
    setup!()

    for action <- [:cancel, :cancel_atomic, :cancel_always_atomic] do
      order = stale_copy_of_paid_order()
      result = update(order, action)
      IO.puts("#{action} on a stale open copy of a paid order: #{result}, stored state: #{inspect(state(order))}")
    end

    # Eight callers confirm the same order at once, each from the open copy they read.
    order = Ash.create!(Repro.Order, %{})

    results =
      1..8
      |> Enum.map(fn _ -> Task.async(fn -> update(order, :confirm) end) end)
      |> Enum.map(&Task.await/1)

    %{rows: [[receipts]]} = Repro.Repo.query!("SELECT count(*) FROM receipts")
    IO.puts("8 concurrent confirm: #{Enum.count(results, &(&1 == "{:ok, _}"))} succeeded, #{receipts} receipts sent")
  end
end

Repro.run()
Expected Behavior

cancel on a stale open copy of a paid order returns NoMatchingTransition and the order stays paid. Of the eight concurrent confirm calls, one succeeds and one receipt is sent.

Maybe the non-atomic path could check the transition against the stored state too, for example by also adding the from-states as a changeset filter (as optimistic_lock does with its version), or by checking again in a before_action after earlier ones? I have a fix along those lines on my fork, which I can open as a PR if that's the direction you want.

Dominant language
Elixir
Stars
37
Forks
16
Avg merge
1h 11m
Merged PRs (30d)
1

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Similar issues

More Elixir issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.