Tasks are running even when orchestration is complete
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- azure, csharp
- Domain
- backend, cloud, distributed-systems
Research direction
Start by tracing the Azure Storage Provider handling of workItem and control queue messages, focusing on lease renewal and deletion around TaskScheduled#5. Compare the logged lifecycle of task #3 with task #5 and determine why the task continues executing after orchestration completion; done means the repeated execution is prevented and the relevant message lifecycle is handled consistently.
Written by the indexing model from the issue text.
Description
Hi,
I am using DTF with Azure Storage Provider.
I have an orchestration which contains several tasks. The orchestration completes successfully. However I see that there are 2 tasks which are hidden in the workItem queue. Even after the orchestration is complete, these tasks keep on executing forever.
I tried to debug from logs to understand what is going on here, but could not go further.
Below is an example of a good task (task#3) and a bad task (task#5) which executes forever.
LifeStory of Task#3
Sending [TaskScheduled#3] message to newprovisioninghub-workitems for instance 'c2102a9a4d5942e4a1bf91e0767b50c8'
Fetched [TaskScheduled#3] message from newprovisioninghub-workitems (delay = 45ms)
Processing [TaskScheduled#3] (total delay = 45ms)
.......... Runs the Task - <>
Sending [TaskCompleted#3] message to newprovisioninghub-control-00 for instance 'c2102a9a4d5942e4a1bf91e0767b50c8'
Deleting [TaskScheduled#3] message from newprovisioninghub-workitems
Fetched [TaskCompleted#3] message from newprovisioninghub-control-00 (delay = 90ms)
Processing [TaskCompleted#3] (total delay = 166ms)
Deleting [TaskCompleted#3] message from newprovisioninghub-control-00
LifeStory of Task#5
Sending [TaskScheduled#5] message to newprovisioninghub-workitems for instance 'c2102a9a4d5942e4a1bf91e0767b50c8'
Fetched [TaskScheduled#5] message from newprovisioninghub-workitems (delay = 0ms)
Processing [TaskScheduled#5] (total delay = 0ms)
newprovisioninghub-control-00: No new messages were found - backing off
Renewing [TaskScheduled#5] message on newprovisioninghub-workitems for an additional 1200 seconds
Sending [TaskCompleted#5] message to newprovisioninghub-control-00 for instance 'c2102a9a4d5942e4a1bf91e0767b50c8'
Deleting [TaskScheduled#5] message from newprovisioninghub-workitems
Fetched [TaskCompleted#5] message from newprovisioninghub-control-00 (delay = 251ms)
Failed to update or delete message [TaskScheduled#5] with ID = c4eb20ac-48be-47d2-bf25-dd202a67c8e4 because it no longer exists
Processing [TaskCompleted#5] (total delay = 335ms)
Deleting [TaskCompleted#5] message from newprovisioninghub-control-00
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 8
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/durabletask
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Azure/durabletask#1398 · 2 comments ·
Maintainers usually reply within 3 days
-
Azure Storage backend: control queue partition left unowned for hours/days after lease expiresMay be free again @nytian claimed this 34 days ago, and no pull request is open. Open
Azure/durabletask#1389 · 1 comment · 1 assignee ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Azure/durabletask#1332 ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 58/100
Azure/durabletask#1318 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Azure/durabletask#1301 · 3 comments ·
Maintainers usually reply within 3 days
All issues in Azure/durabletask
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
PCL-Community/PCL-CE#3652 ·
Maintainers usually reply within 1 day
-
area:frontend bug FE P3
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
klasolsson81/jobbliggaren#2010 ·
Maintainers usually reply within 1 day
-
agentic-workflows untriaged
Difficulty 1/5 Under an hour Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
area: homeblaze type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
RicoSuter/Namotion.Interceptor#630 ·
Maintainers usually reply within 1 day
-
Akka.Hosting enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 65/100