Retrying an interrupted CreateTag never completes the Tag

Open
  • Is something wrong?
#3,386 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
57/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, grpc
Domain
api, backend

Research direction

Start with TagActorSnapshot in cmd/ateapi/internal/controlapi/workflow_tag.go:52-55 and trace the CreateTag retry path for a Tag left in SNAPSHOT_STORAGE_STATE_PENDING. Check the existing CreateTag, GetTag, and DeleteTag behavior, then add or run coverage for retrying after the deadline expires during snapshot copy. Done means the retry converges to a usable completed Tag, with the behavior for an already completed Tag matching the expected response.

Written by the indexing model from the issue text.

Description

area/api area/api-machinery kind/bug
What happened?

A CreateTag call whose deadline expires during the snapshot copy leaves the Tag in SNAPSHOT_STORAGE_STATE_PENDING. Retrying the same request returns AlreadyExists. The Tag stays PENDING until someone deletes it.

TagActorSnapshot documents this (cmd/ateapi/internal/controlapi/workflow_tag.go:52-55): the create is not idempotent, and the retry path is DeleteTag then CreateTag. DeleteTag followed by CreateTag does produce a COMPLETED Tag.

This causes three problems:

  • Clients that retry DeadlineExceeded or Unavailable with the same request never converge. They get AlreadyExists for a Tag they can't use.
  • A Tag left by a failed create looks the same as one still being created. Both are PENDING, so a client can't tell whether to wait or delete. Nothing cleans up the failed one.
Expected Behavior

Retrying the same CreateTag returns the finished Tag, or AlreadyExists for a COMPLETED Tag. No PENDING Tag is left behind.

Steps to Reproduce
  1. Deploy the counter demo: go run ./cmd/ate-setup deploy demo counter.
  2. Create an actor from the counter template. It starts SUSPENDED.
  3. Call CreateTag with source_actor set to that actor and a 20ms gRPC deadline. It returns DeadlineExceeded.
  4. GetTag shows the Tag at version 2 in SNAPSHOT_STORAGE_STATE_PENDING.
  5. Send the same CreateTag with no deadline. It returns AlreadyExists: Tag ate-demo-counter/bb-s1-short-tag already exists; delete it and create it again to retry.
  6. GetTag still shows PENDING. CreateActor with this Tag as source_tag returns FailedPrecondition: source Tag is still being created or failed creation.

kubectl ate can't send a short enough deadline to hit this. A Go client with context.WithTimeout can.

Dominant language
Go
Stars
5.9k
Forks
637
Avg merge
2d 1h
Merged PRs (30d)
523

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.