AzureStorageOrchestrationService - LockNextTaskActivityWorkItem assert failure

Open
#1,173 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
32/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
azure, csharp
Domain
backend, cloud, distributed-systems

Research direction

Start in the OrchestrationService class at activeActivitySessions.TryAdd and trace the surrounding LockNextTaskActivityWorkItem and RenewTaskActivityWorkItemLockAsync paths. Compare this with activity delivery through TaskOrchestrationContext.CallActivityAsync and investigate how the same queue message can be processed concurrently. Done means identifying the cause and a prevention or fix that lets the orchestration continue to the next queued activity.

Written by the indexing model from the issue text.

Description

In one of our orchestrators that is running multiple activities in parallel, we've encountered following warning:

[Warning] Internal assert failure: Work item queue message with ID = {Id} is being processed multiple times concurrently

Activity is being launched via standard TaskOrchestrationContext.CallActivityAsync method. It resulted in orchestrator being stuck in limbo, and after activity ended - not being possible to start next activity from the orchestration queue.

What I have found in the OrchestrationService class, is following piece of code:

                if (!this.activeActivitySessions.TryAdd(message.Id, session))
                {
                    // This means we're already processing this message. This is never expected since the message
                    // should be kept invisible via background calls to RenewTaskActivityWorkItemLockAsync.
                    this.settings.Logger.AssertFailure(
                        this.azureStorageClient.QueueAccountName,
                        this.settings.TaskHubName,
                        $"Work item queue message with ID = {message.Id} is being processed multiple times concurrently.");
                    return null;
                }

According to the comment, this assert should never fail - however it did in our scenario. My question is: any ideas what could be the reason for that, and how to prevent it happening in future?

Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
4d 2h
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Azure/durabletask

All issues in Azure/durabletask

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.