.NET: DurableAIAgent RunAsync polls forever when the agent entity fails before durable state commit (e.g. DurableAgentHistoryBindingMismatchException)

Start in dotnet/src/Microsoft.Agents.AI.DurableTask/AgentEntity.cs around the generic catch near lines 691-722, where the failure is logged and rethrown without committing an outcome. Then read AgentRunHandle.cs ReadAgentOutcomeAsync (lines 58-106) to see why polling never ends. Done means a deterministic binding failure surfaces to RunAsync promptly, verified by a test that sets no ProviderKey and expects an exception rather than a hang.

Description

When the agent entity's Run operation throws before it commits durable state, the caller's
RunAsync never sees the failure. The exception only appears in worker logs. The client keeps
polling entity state with no limit, so RunAsync hangs until the caller cancels it.

Configuration errors hit this path. They are deterministic and will fail on every retry. One example
is a custom ChatHistoryProvider registered without DurableAgentHistoryOptions.ProviderKey.
Schema 2 is now the default (dafx-v2-{agent}, EnablePersistentRequestOutcomes is internal and
true), so apps that worked on earlier 1.16 previews now hang on their first turn after upgrading.

• Repo / commit: main @ 3ffe3fb354b02aba62d482abc829e6014344741f
• Also reproduced with the published Microsoft.Agents.AI.DurableTask 1.16.0-preview.261029.1
  (commit 5289ab6; git diff --stat 5289ab6 3ffe3fb -- dotnet/src is empty)
• Microsoft.Agents.AI 1.23.0, Microsoft.DurableTask.*.AzureManaged 1.18.0, .NET 10, DTS emulator
  (mcr.microsoft.com/dts/dts-emulator:latest)

Steps to reproduce

1. Create a ChatClientAgent with a custom ChatHistoryProvider that stores history externally,
   such as a JSON file. Do not set a provider key:

   AIAgent agent = chatClient.AsAIAgent(new ChatClientAgentOptions
   {
       Name = "BillingAgent",
       ChatHistoryProvider = new FileChatHistoryProvider(path), // StateKeys => ["file-chat-history"]
   });
   
   services.ConfigureDurableAgents(
       options => options.AddAIAgent(agent, timeToLive: TimeSpan.FromHours(1)), // no configureHistory
       workerBuilder: b => b.UseDurableTaskScheduler(cs),
       clientBuilder: b => b.UseDurableTaskScheduler(cs));
   

2. Start the host, resolve the keyed AIAgent, create a session, then call
   await agent.RunAsync("hello", session);.

Observed

The worker logs this error. Event 14 is followed by event 15, "Unhandled exception in entity operation …/Run":

fail: Microsoft.DurableTask.Agents.billingagent.<key>[14]
      [@dafx-v2-billingagent@<key>] Durable agent operation failed before durable state commit.
      Microsoft.Agents.AI.DurableTask.DurableAgentHistoryBindingMismatchException: History ownership 'ExternalProvider' requires an explicit stable logical provider key. Set ProviderKey in the agent registration's history options before running this durable session.
         at Microsoft.Agents.AI.DurableTask.DurableAgentHistoryBinding.RequireProviderKey(DurableAgentHistoryOwnership ownership, String configuredProviderKey, Boolean remoteTransitionDetectedAfterExecution)
         at Microsoft.Agents.AI.DurableTask.DurableAgentHistoryBinding.Create(DurableAgentHistoryOwnership ownership, String configuredProviderKey, Boolean remoteTransitionDetectedAfterExecution)
         at Microsoft.Agents.AI.DurableTask.AgentEntity.Run(RunRequest request)

The client logs Polling agent with session ID ... for response with correlation ID ..., then
returns nothing and throws nothing. RunAsync was still pending when the process was killed after 120 s.

There is a second variant. Setting ProviderKey while the provider leaves its declared StateKeys
unwritten hangs the same way. The failure moves to finalization, after the model call, tool
execution, and the provider's own transcript write have all run:

Microsoft.Agents.AI.DurableTask.DurableAgentHistoryBindingMismatchException: The restored ExternalProvider session does not contain the public continuation evidence required by its fixed durable history binding. The runtime will not initialize a replacement logical conversation; restore the missing continuation or start a new durable session.
   at Microsoft.Agents.AI.DurableTask.DurableAgentHistoryBinding.ValidateBoundContinuation(...)
   at Microsoft.Agents.AI.DurableTask.AgentEntity.FinalizeHistoryStateAsync(...)
   at Microsoft.Agents.AI.DurableTask.AgentEntity.Run(RunRequest request)

In this variant, side effects like the tool call and the external transcript write have already
happened before the caller hangs. That makes the problem hard to diagnose.

Expected

RunAsync should fail promptly with the entity's exception, or a wrapper that carries its message,
when the operation can't succeed. At minimum it should not wait forever. Possible options:

• For deterministic validation and configuration failures, especially pre-execution binding
  validation that runs before any model or provider callback, commit a terminal failed result for
  the correlation ID. The schema 2 mailbox already supports failed outcomes. AgentRunHandle would
  then resolve to a non-Pending outcome and throw.
• Or have AgentRunHandle.ReadAgentOutcomeAsync detect a failed entity operation for its signal
  and stop polling. A configurable maximum wait or timeout would also work.
• Validate ProviderKey and StateKeys at AddAIAgent registration or host startup when a custom
  ChatHistoryProvider can be detected. This catches the first variant before any request is sent.

Code pointers (@3ffe3fb)

• dotnet/src/Microsoft.Agents.AI.DurableTask/AgentEntity.cs:691-722: the generic catch logs
  LogDurableAgentExecutionFailed and rethrows (throw; at line 721). Nothing is committed for the
  correlation ID unless TryFinalizeProviderFailureAsync applies.
• dotnet/src/Microsoft.Agents.AI.DurableTask/AgentRunHandle.cs:58-106: ReadAgentOutcomeAsync
  has a while (true) loop that exits only on a non-Pending outcome or a cancelled token. The
  backoff is capped at 3 s and there's no overall limit.
• dotnet/src/Microsoft.Agents.AI.DurableTask/DurableAIAgentProxy.cs:91-99: the proxy signals the
  entity (RunAgentAsync), then awaits ReadAgentResponseAsync(cancellationToken). A failure of
  the signalled operation never reaches the caller.
• dotnet/src/Microsoft.Agents.AI.DurableTask/DurableAgentHistoryBinding.cs:416-430
  (RequireProviderKey) and :295-322 (ValidateBoundContinuation) throw the two exceptions above.
• dotnet/src/Microsoft.Agents.AI.DurableTask/Logs.cs:127-134: event 14, "failed before durable
  state commit".

Workaround

Pass a CancellationToken with a timeout to RunAsync, and check worker logs for event 14.
To fix the configuration itself, set
configureHistory: h => h.ProviderKey = new("your.history.v1") and have the provider write a
non-empty value for each declared StateKey into session.StateBag.

Related

• #143 (external history-provider failures and at-least-once execution) covers the side-effect
  semantics, not the hang.
• #106 (durable history ownership transitions).