Parsing errors while DTFx processes the messages from queue in latest version of DurableTask.AzureStorage 2.0.0-rc.2.

Open
#1,141 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
azure, csharp

Research direction

Start with the stack trace locations in src/DurableTask.AzureStorage/Utils.cs, MessageManager.cs, and Messaging/WorkItemQueue.cs, focusing on how queue messages are deserialized after upgrading to 2.0.0-rc.2. Compare the message format and serialization behavior with the previous version; done means valid queue messages are processed without JsonReaderException and scheduled workflows execute reliably.

Written by the indexing model from the issue text.

Description

Hii,
We started consuming the 2.0.0-rc.2 version of DurableTask.AzureStorage nuget as previous versions contain reference of the deprecated versions of "WindowsAzure.Storage".

After consuming the nuget we deployed in the ECY Canary region for testing. After deployment, we started encountering the Json parsing issues while dtfx processes the queue messages. "Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: e. Path '', line 0, position 0.".

Full exception message with stack trace:

An error occurred while processing messages on springboot-workitems: Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: e. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at DurableTask.AzureStorage.Utils.DeserializeFromJson[T](JsonSerializer serializer, String jsonString) in /_/src/DurableTask.AzureStorage/Utils.cs:line 202
   at DurableTask.AzureStorage.MessageManager.DeserializeQueueMessageAsync(QueueMessage queueMessage, String queueName, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 176
   at DurableTask.AzureStorage.Messaging.WorkItemQueue.GetMessageAsync(CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Messaging/WorkItemQueue.cs:line 48

We started encountering these issues after updating to the "2.0.0-rc.2", prior to that we didn't observe these issues. We observed that a periodic workflow which we run every 5 mins is not getting executed but there are few successful executions of other workflows as well.

Let me know if you need anything further to debug this issue.

Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
4d 2h
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Azure/durabletask

All issues in Azure/durabletask

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.