Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[WORKFLOW SDK FEATURE REQUEST] Retry WaitForInstanceCompletion/Start on a server-sent CANCELLED

Open
  • Is something wrong?
#1,246 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend, distributed-systems

Research direction

Start in dapr/ext/workflow/_durabletask/client.py around lines 335-349 and the matching retry logic in aio/client.py. Read the three cancellation tests in tests/ext/workflow/durabletask/test_orchestration_wait.py and test_client_async.py, then confirm the intended retry and timeout behavior with maintainers. Done means both clients re-issue eligible waits after server CANCELLED responses while preserving caller deadlines and the updated tests pass.

Written by the indexing model from the issue text.

Description

dapr-ext-workflow kind/enhancement

Proposal

wait_for_workflow_completion and wait_for_workflow_start (sync and aio) should re-issue WaitForInstanceCompletion / WaitForInstanceStart when the server ends the call with CANCELLED and the caller's own timeout hasn't expired. Today that status reaches the caller as an error, even though the workflow is still running. This asks maintainers to decide on the behaviour first, because it reverses assertions added in #1112. It's a proposal, not a PR.

Why the server sends CANCELLED

When daprd shuts down or restarts mid-wait (rollout, pod restart, hot-reload, a config change that restarts the runtime), the actors router cancels every in-flight call it tracks. The wait then returns CANCELLED "context canceled", even though the caller is still waiting. The mechanism and a repro are in dapr/dapr#10566.

Reproduced against native daprd 1.18.0:

  • a 60 s workflow with wait_for_workflow_completion(id) and no timeout returns normally;
  • the same run with daprd sent SIGTERM about 10 s in fails with StatusCode.CANCELLED "context canceled".

Why retrying is safe

  • A blocking unary call only gets CANCELLED from the server. The sync client can't cancel it mid-flight, and in the aio client a caller's cancellation raises asyncio.CancelledError, not AioRpcError. A client-side deadline shows up as DEADLINE_EXCEEDED, which already maps to TimeoutError.
  • The wait only reads state, so re-issuing it is idempotent. It returns straight away if the workflow already finished.

What would change

Relationship to the runtime fix

The proper fix is in daprd: return UNAVAILABLE in this case (dapr/dapr#10566). The SDK already retries UNAVAILABLE. This change would protect users on current and older runtimes until that ships, and it would still help afterwards with proxies that reset the stream.

Question for maintainers

Is it acceptable to treat a server-sent CANCELLED on these two waits as retryable, and change the #1112 tests? If yes, it's a small change in both clients plus tests.

Dominant language
Python
Stars
272
Forks
152
Avg merge
2d 22h
Merged PRs (30d)
7

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dapr/python-sdk

All issues in dapr/python-sdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.