[Feature Request] Add opt-in retry policy for transient LLM failures

Add opt-in retry policy for transient LLM failures

Please make sure you read the contribution guide and file the issues in the right place.
Contribution guide

🔴 Required Information

Is your feature request related to a specific problem?

Yes.

The Java ADK Runner currently does not provide a configurable retry mechanism for transient LLM provider failures such as rate limiting, service unavailability, network timeouts, or connection failures.

Applications must either fail the invocation immediately or retry the entire Runner.runAsync(...) call. Retrying the complete invocation is unsafe because the runner may have already:

• Appended the user message to the session.
• Updated session state.
• Emitted or persisted model events.
• Executed tools with external side effects.

A whole-invocation retry can therefore duplicate session events or repeat non-idempotent tool operations.

Describe the Solution You'd Like

Add an opt-in retry policy at the LLM model-call boundary inside BaseLlmFlow.

The policy should wrap BaseLlm.generateContent(...), rather than retrying the complete runner invocation. This would allow transient provider failures to be retried without replaying user messages, tools, or other runner-level side effects.

The proposed behavior is:

• Retry is disabled by default to preserve existing behavior.
• Retry transient provider and network failures only.
• Use configurable bounded exponential backoff with optional jitter.
• Count every provider attempt against RunConfig.maxLlmCalls().
• Never retry after a streaming response has emitted its first LlmResponse.
• Delegate provider-specific exception classification to each BaseLlm implementation.
• Record retry attempts in logs and tracing attributes.

Impact on your work

We use ADK Java for event-driven agent workflows where temporary rate limits, provider outages, and network interruptions should not immediately fail an otherwise recoverable invocation.

Runner-level retries are not suitable because our agents can update session history and invoke tools with external side effects. Retrying only the model call would improve resilience while preserving session consistency and avoiding duplicate operations.

This feature would also let us enforce a predictable cost limit because every failed and successful provider attempt would count toward maxLlmCalls().

Timeline: N/A.

Willingness to contribute

Yes. We are willing to implement this feature and submit a PR.

---

🟡 Recommended Information

Describe Alternatives You've Considered

1. Retrying Runner.runAsync(...)

   This can duplicate user events, session mutations, model events, and tool executions because the runner appends the user message before agent execution begins.

2. Retrying in an application-specific agent wrapper

   This avoids modifying ADK but does not provide a reusable solution for other ADK Java users. It also operates at too broad a boundary and can repeat agent-side effects.

3. Relying entirely on provider SDK retries

   Provider behavior varies, is not consistently configurable across supported LLMs, and may not integrate with ADK's maxLlmCalls() cost limit or tracing.

4. Retrying all exceptions

   This could repeatedly submit invalid prompts, schemas, credentials, or model configurations. Retry decisions should remain provider-specific and limited to transient failures.

Proposed API / Implementation

Add an opt-in retry configuration to RunConfig:

RunConfig runConfig =
    RunConfig.builder()
        .maxLlmCalls(12)
        .retryConfig(
            RetryConfig.builder()
                .maxAttempts(3)
                .initialBackoff(Duration.ofMillis(500))
                .maxBackoff(Duration.ofSeconds(8))
                .multiplier(2.0)
                .jitterRatio(0.2)
                .retryableStatusCodes(Set.of(408, 429, 500, 502, 503, 504))
                .build())
        .build();

The default configuration would disable retries:

RetryConfig.disabled(); // maxAttempts = 1

Add a provider-specific classification method to BaseLlm:

public abstract boolean isExceptionRetryable(
    Throwable exception,
    Set<Integer> retryableStatusCodes);

Each implementation would classify its own SDK exceptions:

public class Gemini extends BaseLlm {
  @Override
  public boolean isExceptionRetryable(
      Throwable exception,
      Set<Integer> retryableStatusCodes) {
    return exception instanceof ApiException apiException
            && retryableStatusCodes.contains(apiException.code())
        || exception instanceof GenAiIOException
        || exception instanceof IOException
        || exception instanceof TimeoutException;
  }
}

Equivalent implementations would be provided for OpenAI, Groq, Claude, and Apigee.

BaseLlmFlow would delegate the provider call to a shared retry policy:

return LlmRetryPolicy.execute(
    context,
    llm,
    finalLlmRequest,
    context.runConfig().streamingMode() == StreamingMode.SSE);

The shared policy would:

1. Increment the LLM call count before each real provider attempt.
2. Call BaseLlm.generateContent(...).
3. Traverse the exception cause chain safely.
4. Ask llm.isExceptionRetryable(...) to classify each cause.
5. Apply bounded exponential backoff and jitter.
6. Stop after maxAttempts.
7. Stop immediately after any streaming response has been emitted.
8. Propagate LlmCallsLimitExceededException without retrying.

Example classification flow:

static boolean isRetryable(
    BaseLlm llm,
    Throwable error,
    RetryConfig config) {
  Set<Throwable> visited =
      Collections.newSetFromMap(new IdentityHashMap<>());

  for (Throwable current = error;
      current != null && visited.add(current);
      current = current.getCause()) {

    if (current instanceof LlmCallsLimitExceededException) {
      return false;
    }

    if (llm.isExceptionRetryable(
        current, config.retryableStatusCodes())) {
      return true;
    }
  }

  return false;
}

Additional Context

The intended retry boundary is:

Runner
  → append user event once
  → agent.runAsync(...)
  → BaseLlmFlow
  → LlmRetryPolicy
  → BaseLlm.generateContent(...)

Failed attempts would not produce ADK events or enter session history.

Suggested acceptance criteria:

• Existing callers experience no behavior change unless retries are enabled.
• Transient provider failures are retried with bounded backoff.
• Non-retryable client, authentication, and configuration errors fail immediately.
• Failed attempts do not create persisted model events.
• User messages and tool calls are not replayed.
• Every provider attempt consumes the maxLlmCalls() budget.
• Streaming calls retry only when no response has been emitted.
• Retry attempts include model, agent, attempt number, delay, and error type in logs or traces.
• Unit tests cover disabled retries, success after retry, non-retryable failures, exhausted attempts, call-budget exhaustion, and partial-stream failures.