AzureStorageOrchestrationService fails to store an event once OrchestrationStatus gets bigger

Open
#1,185 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
azure, csharp

Research direction

Start at AzureTableTrackingStore.cs line 992 and follow the entity update through Table.cs line 103 and AzureStorageOrchestrationService.cs line 1202. Reproduce the failure with an orchestration status larger than Azure Table Storage permits, then compare the Service Fabric and Emulator behavior noted in the issue. Done means Azure Storage can persist the affected orchestration without the PropertyValueTooLarge failure.

Written by the indexing model from the issue text.

Description

If the orchestration status is a more complex class, that serializes into a larger string, the following exception is thrown and orchestration does not proceed further:

DurableTask.Core[29]
TaskOrchestrationDispatcher-091002dd69a645bb89247b8f24b44b98-0: Unhandled exception with work item '8652685d247d44589559f13e3096234e': DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
 ---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge

Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}

Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8

   at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
   at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
   at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

Backing off for 10 seconds until 5 successful operations
DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
 ---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge

Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}

Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8

   at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
   at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
   at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

This limitation is not present for ServiceFabric and Emulator.

Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
4d 2h
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Azure/durabletask

All issues in Azure/durabletask

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.