AzureStorageOrchestrationService fails to store an event once OrchestrationStatus gets bigger
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
Research direction
Start at AzureTableTrackingStore.cs line 992 and follow the entity update through Table.cs line 103 and AzureStorageOrchestrationService.cs line 1202. Reproduce the failure with an orchestration status larger than Azure Table Storage permits, then compare the Service Fabric and Emulator behavior noted in the issue. Done means Azure Storage can persist the affected orchestration without the PropertyValueTooLarge failure.
Written by the indexing model from the issue text.
Description
If the orchestration status is a more complex class, that serializes into a larger string, the following exception is thrown and orchestration does not proceed further:
DurableTask.Core[29]
TaskOrchestrationDispatcher-091002dd69a645bb89247b8f24b44b98-0: Unhandled exception with work item '8652685d247d44589559f13e3096234e': DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge
Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}
Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
--- End of inner exception stack trace ---
at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373
Backing off for 10 seconds until 5 successful operations
DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge
Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}
Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
--- End of inner exception stack trace ---
at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373
This limitation is not present for ServiceFabric and Emulator.
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 6
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/durabletask
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Azure/durabletask#1398 · 2 comments ·
-
Azure/durabletask#1389 · 1 comment · 1 assignee ·
-
Difficulty 3/5 1-2 days Newbie friendliness 74/100
Azure/durabletask#1376 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
Azure/durabletask#1378 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Azure/durabletask#1332 ·
All issues in Azure/durabletask
Similar issues
-
[Feat] 조합 영역 구분선 개선 Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
type/automation type/tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
t/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
ci-failure-cause test-failure
Difficulty 2/5 1-3 hours Newbie friendliness 82/100