getResult(timeout) busy-loops GetWorkflowExecutionHistory in the last second before the timeout (server 1.29+)

Start with WorkflowClientLongPollHelper.getInstanceCloseEvent and WorkflowClientLongPollAsyncHelper.getInstanceCloseEventAsync, identified in the issue, and trace how empty poll responses are retried against the shared deadline. The issue proposes a minimum interval between empty polls for both paths and a fake-client unit test that mimics the server soft timeout. Done means the test passes, requests are paced without exceeding the deadline, and the sync and async paths are covered.

When WorkflowStub.getResult(timeout, unit, ...) (or getResultAsync) waits on a workflow that doesn't complete within the timeout, the SDK re-sends GetWorkflowExecutionHistory back to back, with no delay, for the last second before the deadline. Against a local dev server, a single 2s wait that times out sends roughly 3,000-4,000 history requests. The same code sends 1 request against server 1.28.0, so this looks like a regression that started with server 1.29.0.

With several callers waiting at the same time, the burst also hits the namespace rate limit. getResult then fails with RESOURCE_EXHAUSTED instead of throwing TimeoutException, and it returns later than the requested timeout.

Expected Behavior

getResult(2, TimeUnit.SECONDS, ...) on a workflow that is still running should wait with one or two history long polls and throw TimeoutException after 2 seconds. This is what happens against server 1.28.0 (1 request).

Actual Behavior

GetWorkflowExecutionHistory calls for a single getResult(2s) that times out on a running workflow:

• The calls are packed into the last second before the deadline, so the timeout value barely changes the count. 1s, 5s and 25s waits produced about 2,500-4,600 calls each.
• getResultAsync behaves the same way (~3,800-4,000 calls).

With concurrent waiters (server 1.32.0, N threads each calling getResult(2s) twice on the same running workflow):

The failed calls got a WorkflowServiceException caused by StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded after 2.0-2.9 seconds. In one run, a call failed with service rate limit exceeded instead. Other calls in the same namespace (DescribeWorkflowExecution and a non-waiting GetWorkflowExecutionHistory) were not throttled in these runs.

We also saw this in production on a self-hosted cluster: every short getResult wait that timed out produced a few hundred history requests. It went away after we replaced getResult with describe() and a retry from the client.

Steps to Reproduce the Problem

1. Start a dev server with Temporal CLI 1.9.1 (server 1.32.0): temporal server start-dev
2. Run the program below with Java SDK 1.40.0.
3. It prints something like getResult(2s) timed out after 2004 ms, GetWorkflowExecutionHistory calls: 2981.
4. Against a dev server from Temporal CLI 1.4.1 (server 1.28.0), the same program prints calls: 1.

import io.grpc.CallOptions;
import io.grpc.Channel;
import io.grpc.ClientCall;
import io.grpc.ClientInterceptor;
import io.grpc.MethodDescriptor;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.client.WorkflowStub;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.worker.WorkerFactory;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import java.time.Duration;
import java.util.Collections;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;

public class GetResultHistoryFloodRepro {

  @WorkflowInterface
  public interface SleepWorkflow {
    @WorkflowMethod
    void run();
  }

  public static class SleepWorkflowImpl implements SleepWorkflow {
    @Override
    public void run() {
      Workflow.sleep(Duration.ofMinutes(5));
    }
  }

  public static void main(String[] args) throws Exception {
    AtomicInteger historyCalls = new AtomicInteger();
    ClientInterceptor countHistoryCalls =
        new ClientInterceptor() {
          @Override
          public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
              MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
            if ("GetWorkflowExecutionHistory".equals(method.getBareMethodName())) {
              historyCalls.incrementAndGet();
            }
            return next.newCall(method, callOptions);
          }
        };

    WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                .setTarget("127.0.0.1:7233")
                .setGrpcClientInterceptors(Collections.singletonList(countHistoryCalls))
                .build());
    WorkflowClient client = WorkflowClient.newInstance(service);
    WorkerFactory factory = WorkerFactory.newInstance(client);
    factory.newWorker("repro").registerWorkflowImplementationTypes(SleepWorkflowImpl.class);
    factory.start();

    SleepWorkflow workflow =
        client.newWorkflowStub(
            SleepWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue("repro")
                .setWorkflowId("repro-" + System.currentTimeMillis())
                .build());
    WorkflowClient.start(workflow::run);
    WorkflowStub stub = WorkflowStub.fromTyped(workflow);
    Thread.sleep(1000);

    historyCalls.set(0);
    long startedAt = System.nanoTime();
    try {
      stub.getResult(2, TimeUnit.SECONDS, Void.class);
    } catch (TimeoutException expected) {
      // The workflow sleeps for 5 minutes, so the wait times out.
    }
    System.out.printf(
        "getResult(2s) timed out after %d ms, GetWorkflowExecutionHistory calls: %d%n",
        (System.nanoTime() - startedAt) / 1_000_000, historyCalls.get());

    stub.terminate("repro done");
    factory.shutdownNow();
    service.shutdownNow();
    System.exit(0);
  }
}

Specifications

• Version: Java SDK 1.40.0 (also reproduced with 1.38.0)
• Platform: macOS (Darwin 25.4.0, arm64), OpenJDK 25.0.2
• Server: Temporal CLI dev server 1.4.1 (server 1.28.0, not affected), 1.6.1 (server 1.30.1) and 1.9.1 (server 1.32.0)
• Regression: works as expected up to server 1.28.0. Starts with 1.29.0, which includes temporalio/temporal#8238.

Root cause

Since server 1.29.0, a GetWorkflowExecutionHistory long poll only waits until one second before the caller's deadline and then returns an empty response (temporalio/temporal#8238, "GetWorkflowExecutionHistory long poll soft timeout"). Once less than one second is left, the computed wait is zero and the server returns an empty response right away:

• https://github.com/temporalio/temporal/blob/e161e3bc28d4e86ce76ae5f83d12c2443e76cb8d/service/history/api/get_workflow_util.go#L190-L191
• https://github.com/temporalio/temporal/blob/e161e3bc28d4e86ce76ae5f83d12c2443e76cb8d/common/contextutil/deadline.go#L34-L36
• https://github.com/temporalio/temporal/blob/e161e3bc28d4e86ce76ae5f83d12c2443e76cb8d/common/constants.go#L36-L39

On the SDK side, WorkflowClientLongPollHelper.getInstanceCloseEvent uses the same overall deadline for every poll and sends the next request immediately after an empty response. getInstanceCloseEventAsync does the same:

• https://github.com/temporalio/sdk-java/blob/705c3bbe6685eb05c2d60294c8d9c05578e23dbe/temporal-sdk/src/main/java/io/temporal/internal/client/WorkflowClientLongPollHelper.java#L70-L113
• https://github.com/temporalio/sdk-java/blob/705c3bbe6685eb05c2d60294c8d9c05578e23dbe/temporal-sdk/src/main/java/io/temporal/internal/client/WorkflowClientLongPollAsyncHelper.java#L91-L101

Together, these turn the last second into a loop of "empty response, retry right away" until the deadline. The server change was meant to reduce connection churn, but when the remaining deadline is short it multiplies the number of requests instead.

Possible fix

In the SDK, keep a minimum interval (e.g. 100ms) between polls when the previous poll came back empty, without going past the deadline. Normal long polls, which return after several seconds, are not affected.

Results from a local test with this change applied to the sync path only:

The async path needs the same rule. Since the SDK still targets Java 8, I'd like to know the preferred way to schedule the delay there (for example, reusing an existing scheduler).

Another option is to change the server so that it doesn't answer immediately when the remaining time is below the buffer. That would cover every SDK, but it's a server change.

Workaround

Don't rely on getResult(timeout) timing out while handling a request. Check the status with describe() first and call getResult only for completed executions, or let the client retry. This is what we ended up doing.

If this direction sounds reasonable, I'm happy to open a PR with the sync and async changes and a unit test. The test uses a fake client that mimics the server's soft timeout; it fails on the current code and passes with the change.