Python: [Bug]: MAF workflow terminates on transient Foundry exceptions instead of retrying — workflow instance cannot be reused after crash

Description

## Bug summary

When Azure AI Foundry returns a transient exception (e.g. `ReadError`, `APIError` 5xx, `httpx.RemoteProtocolError`), MAF terminates the entire workflow instead of retrying the failing agent step from the last checkpoint. Foundry's own error body explicitly says *"You can retry your request"* — the framework does not act on this.

Additionally, after any such crash the workflow instance keeps its internal `_is_running` flag set to `True`, so calling `.run()` on the same instance again raises `RuntimeError: Workflow is already running. Concurrent executions are not allowed.`

## Severity

High — production impact, intermittent, user-visible failure.

## Component

- `agent_framework` — `Workflow.run()` / `AgentExecutor`
- `agent_framework_foundry._agent._FoundryAgentChatClient`
- Affects streaming (`stream=True`) and non-streaming paths
- Affects Foundry-hosted MCP and client-side streamable HTTP MCP agents

## Why this is a bug

1. The framework already has checkpoint infrastructure that captures state after every superstep.
2. The framework already owns the streaming event pump and knows exactly which agent failed.
3. The exceptions raised are documented by Foundry as retryable.
4. The framework does not honour Foundry's retry guidance.
5. The `finally` cleanup that should reset `_is_running` doesn't complete on the exception path.

## Actual behaviour

Any of the following transient errors terminates the entire workflow, discarding in-memory progress of prior successful agents:

### Error A — Transport-level disconnect (frequent)

```
agent_framework.exceptions.ChatClientException: (
    "<class 'agent_framework_foundry._agent._FoundryAgentChatClient'> service failed to complete the prompt: ",
    ReadError('')
)
```

### Error B — Foundry-side 5xx (Foundry explicitly says "retry")

```
<class 'agent_framework_foundry._agent._FoundryAgentChatClient'> service failed to complete the prompt:
The server had an error processing your request. Sorry about that! You can retry your request, or contact
us through an Azure support request at: https://go.microsoft.com/fwlink/?linkid=2213926 if you keep seeing
this error. (Please include the request ID ****** in your email.)",
APIError('The server had an error processing your request. ...')
```

Foundry itself says **"You can retry your request"** in the error body. MAF does not retry.



## Expected behaviour

1. On a caught exception from a Foundry / `_FoundryAgentChatClient` call, MAF classifies it as transient (`ReadError`, `APIError`, `RemoteProtocolError`, `APIConnectionError`, `APIStatusError` 5xx, `TimeoutError`).
2. MAF loads the latest checkpoint under the workflow's name (already persisted).
3. MAF re-runs the failing agent step from that checkpoint — **without re-executing prior successful agents**.
4. Retry up to a configurable `max_agent_retry_attempts` with configurable backoff.
5. Only after retries are exhausted, propagate the exception to the caller.


## Reproducer

Errors A and B occur sporadically under normal production load — no deterministic reproducer, since they are transient by nature.

```

## Impact

- Every Foundry transient blip surfaces as a full user-visible workflow failure with no automatic recovery.
- Every embedding application ends up implementing its own retry loop around `workflow.run()` — retries that MAF could handle more precisely, because only MAF knows the correct superstep boundary to rewind to.
- Foundry's own error message recommends retry; the framework doesn't act on that guidance.

## Ask
 **Add built-in agent-level retry** on transient Foundry exceptions. MAF already owns checkpoints, streaming, and executor identity — this is where retry logically belongs.


## Related

- Foundry error body itself recommends retry.
- Similar retry-on-transient behaviour is standard in other Microsoft SDKs (Azure SDK, OpenAI SDK auto-retry on 5xx).

Code Sample

## Code sample from our codebase

### 1. Build the workflow

python
from agent_framework import (
    AgentExecutor, Case, Default, Executor, WorkflowBuilder,
)
from agent_framework.foundry import FoundryAgent

def build_workflow(
    agent_list, credential, *,
    checkpoint_storage, workflow_name=None, mcp_functions=None, ...
):
    # Resolve agent_list → AgentExecutor per task
    agent_executors = {}
    for executor_id, agent_name, task_name in resolved:
        foundry = FoundryAgent(
            project_endpoint=ENDPOINT,
            agent_name=agent_name,
            credential=credential,
            tools=tools if use_stream else None,   # client-side streamable MCP for long-running
        )
        agent_executors[executor_id] = AgentExecutor(
            foundry, id=executor_id,
            context_mode="custom",
            context_filter=_clean_messages,
        )

    builder = WorkflowBuilder(
        name=workflow_name,                        # stable = wf.message_id, for checkpoint resume
        start_executor=agent_executors[regular[0][0]],
        checkpoint_storage=checkpoint_storage,
        output_from=[summary_executor],
        intermediate_output_from="all_other",
    )

    for i, (executor_id, _name, _task) in enumerate(regular):
        agent_exec     = agent_executors[executor_id]
        success_target = agent_executors[regular[i + 1][0]] if i < len(regular) - 1 else summary_executor
        builder.add_switch_case_edge_group(
            agent_exec,
            [
                Case(condition=_is_agent_failure, target=summary_executor),
                Default(target=success_target),
            ],
        )

    return builder.build(), executor_info, summary_foundry

### 2. Run the workflow (where the failure surfaces)

python
async def run_pipeline_step(
    workflow, *,
    initial_message: str | None = None,
    checkpoint_id: str | None = None,
    checkpoint_storage,
    on_event=None,
    fallback_summary_agent=None,
) -> dict:
    # ... per-executor chunk buffers, _flush_buffer, _append_chunk, etc. omitted ...

    try:
        if checkpoint_id:
            stream = workflow.run(
                checkpoint_id=checkpoint_id,
                checkpoint_storage=checkpoint_storage,
                stream=True,
            )
        else:
            stream = workflow.run(initial_message, stream=True)

        # Drain the workflow event stream
        async for event in stream:
            etype = getattr(event, "type", None)
            if etype == "intermediate":
                await _append_chunk(event.data, getattr(event, "executor_id", ""))
            elif etype == "output":
                await _flush_buffer()
                _append_terminal(event.data)
        await _flush_buffer()
        await _flush_terminal()
        result = await stream.get_final_response()

    except Exception as exc:
        # ← THIS is where the intermittent Foundry errors land:
        #     ChatClientException wrapping ReadError('')
        #     ChatClientException wrapping APIError('The server had an error ... You can retry your request')
        #     httpx.RemoteProtocolError: Server disconnected without sending a response
        error_msg = f"{type(exc).__name__}: {exc}"
        logger.error(f"\n  [pipeline_step] Exception: {error_msg}")
        logger.error(traceback.format_exc())
        await _emit(on_event, "pipeline_failed", {"error": error_msg})
        return await _run_fallback_summary(error_msg, fallback_summary_agent, initial_message, on_event)

Error Messages / Stack Traces

MAF pipeline failed ...service failed to complete the prompt... ReadError('')

Package Versions

agent-framework-core 1.8.0 , agent-framework-foundry 1.8.0 , agent-framework-orchestrations 1.0.0rc3

Python Version

Python 3.11

Additional Context

No response