Out of Memory Exception when Activity Trigger Function is called.

Start with the referenced src/DurableTask.AzureStorage/Messaging/WorkItemQueue.cs and follow the stack trace through MessageManager.cs and Utils.cs. Reproduce the large-payload failure using the package versions and Application Insights error described in the issue. Done means the team has an agreed, tested handling or documented limit for this message path.

Hello,
I am running into an issue where my Azure Durable Function run under Azure Resource with In-Process Model. My application uses the following versions of the packages

Microsoft.Azure.WebJobs.Extensions.DurableTask 2.10.0
Microsoft.Azure.DurableTask.AzureStorage 1.14.0
Microsoft.Azure.DurableTask.Core 2.13.0
Microsoft.Azure.WebJobs.Extensions.DurableTask.Analyzers 0.5.0
Microsoft.DurableTask.Sidecar.Protobuf 1.0.0

My Orchestrator calls an activity which calls another activity with a huge array in the parameter, and that last activity function never runs and I see an OutOfMemoryException in the Application Insights logs (stack trace at the end).

I have looked at the relevant code in this repo and it looks like the storage queue has that array saved as JSON and the activity function fails to even get triggered because the upstream logic that puts the array back together from JSON fails. Please correct me if this is wrong since I am not very familiar with the business logic of Durable Task Framework.

My questions for this team are

• Will upgrading to the latest version of Durable Task Framework packages solve my issue?
• I am planning on saving the JSON into a blob and passing the activity trigger function with a SAS URL of the blob, and do my own deserialization inside the activity function, how does this idea sound?
• How to best diagnose this issue, any guide with the setup instructions for the dev environment would be nice

An error occurred while processing messages on functionsappname-workitems: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)
   at Newtonsoft.Json.JsonTextReader.ParseString(Char quote, ReadType readType)
   at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)
   at Newtonsoft.Json.JsonTextReader.ReadAsString()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at DurableTask.AzureStorage.Utils.DeserializeFromJson[T](JsonSerializer serializer, String jsonString) in /_/src/DurableTask.AzureStorage/Utils.cs:line 200
   at DurableTask.AzureStorage.MessageManager.DeserializeQueueMessageAsync(QueueMessage queueMessage, String queueName) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 148
   at DurableTask.AzureStorage.Messaging.WorkItemQueue.GetMessageAsync(CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Messaging/WorkItemQueue.cs:line 47