Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

getResult(timeout) busy-loops GetWorkflowExecutionHistory in the last second before the timeout (server 1.29+)

Offen
  • Stimmt etwas nicht?
#3,132 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
55/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
java
Bereich
backend-api-design

Rechercherichtung

Start with WorkflowClientLongPollHelper.getInstanceCloseEvent and WorkflowClientLongPollAsyncHelper.getInstanceCloseEventAsync, identified in the issue, and trace how empty poll responses are retried against the shared deadline. The issue proposes a minimum interval between empty polls for both paths and a fake-client unit test that mimics the server soft timeout. Done means the test passes, requests are paced without exceeding the deadline, and the sync and async paths are covered.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

When WorkflowStub.getResult(timeout, unit, ...) (or getResultAsync) waits on a workflow that doesn't complete within the timeout, the SDK re-sends GetWorkflowExecutionHistory back to back, with no delay, for the last second before the deadline. Against a local dev server, a single 2s wait that times out sends roughly 3,000-4,000 history requests. The same code sends 1 request against server 1.28.0, so this looks like a regression that started with server 1.29.0.

With several callers waiting at the same time, the burst also hits the namespace rate limit. getResult then fails with RESOURCE_EXHAUSTED instead of throwing TimeoutException, and it returns later than the requested timeout.

Expected Behavior

getResult(2, TimeUnit.SECONDS, ...) on a workflow that is still running should wait with one or two history long polls and throw TimeoutException after 2 seconds. This is what happens against server 1.28.0 (1 request).

Actual Behavior

GetWorkflowExecutionHistory calls for a single getResult(2s) that times out on a running workflow:

Server SDK GetWorkflowExecutionHistory calls
1.28.0 1.40.0 1
1.30.1 1.38.0 ~3,800-4,300
1.32.0 1.40.0 ~2,900-4,000
  • The calls are packed into the last second before the deadline, so the timeout value barely changes the count. 1s, 5s and 25s waits produced about 2,500-4,600 calls each.
  • getResultAsync behaves the same way (~3,800-4,000 calls).

With concurrent waiters (server 1.32.0, N threads each calling getResult(2s) twice on the same running workflow):

Concurrent callers Threw TimeoutException Failed with RESOURCE_EXHAUSTED
3 0 of 6 6 of 6
10 1 of 20 19 of 20
20 4 of 40 36 of 40

The failed calls got a WorkflowServiceException caused by StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded after 2.0-2.9 seconds. In one run, a call failed with service rate limit exceeded instead. Other calls in the same namespace (DescribeWorkflowExecution and a non-waiting GetWorkflowExecutionHistory) were not throttled in these runs.

We also saw this in production on a self-hosted cluster: every short getResult wait that timed out produced a few hundred history requests. It went away after we replaced getResult with describe() and a retry from the client.

Steps to Reproduce the Problem

  1. Start a dev server with Temporal CLI 1.9.1 (server 1.32.0): temporal server start-dev
  2. Run the program below with Java SDK 1.40.0.
  3. It prints something like getResult(2s) timed out after 2004 ms, GetWorkflowExecutionHistory calls: 2981.
  4. Against a dev server from Temporal CLI 1.4.1 (server 1.28.0), the same program prints calls: 1.
Repro (single file, public APIs only) 
import io.grpc.CallOptions;
import io.grpc.Channel;
import io.grpc.ClientCall;
import io.grpc.ClientInterceptor;
import io.grpc.MethodDescriptor;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.client.WorkflowStub;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.worker.WorkerFactory;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import java.time.Duration;
import java.util.Collections;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;

public class GetResultHistoryFloodRepro {

  @WorkflowInterface
  public interface SleepWorkflow {
    @WorkflowMethod
    void run();
  }

  public static class SleepWorkflowImpl implements SleepWorkflow {
    @Override
    public void run() {
      Workflow.sleep(Duration.ofMinutes(5));
    }
  }

  public static void main(String[] args) throws Exception {
    AtomicInteger historyCalls = new AtomicInteger();
    ClientInterceptor countHistoryCalls =
        new ClientInterceptor() {
          @Override
          public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
              MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
            if ("GetWorkflowExecutionHistory".equals(method.getBareMethodName())) {
              historyCalls.incrementAndGet();
            }
            return next.newCall(method, callOptions);
          }
        };

    WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                .setTarget("127.0.0.1:7233")
                .setGrpcClientInterceptors(Collections.singletonList(countHistoryCalls))
                .build());
    WorkflowClient client = WorkflowClient.newInstance(service);
    WorkerFactory factory = WorkerFactory.newInstance(client);
    factory.newWorker("repro").registerWorkflowImplementationTypes(SleepWorkflowImpl.class);
    factory.start();

    SleepWorkflow workflow =
        client.newWorkflowStub(
            SleepWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue("repro")
                .setWorkflowId("repro-" + System.currentTimeMillis())
                .build());
    WorkflowClient.start(workflow::run);
    WorkflowStub stub = WorkflowStub.fromTyped(workflow);
    Thread.sleep(1000);

    historyCalls.set(0);
    long startedAt = System.nanoTime();
    try {
      stub.getResult(2, TimeUnit.SECONDS, Void.class);
    } catch (TimeoutException expected) {
      // The workflow sleeps for 5 minutes, so the wait times out.
    }
    System.out.printf(
        "getResult(2s) timed out after %d ms, GetWorkflowExecutionHistory calls: %d%n",
        (System.nanoTime() - startedAt) / 1_000_000, historyCalls.get());

    stub.terminate("repro done");
    factory.shutdownNow();
    service.shutdownNow();
    System.exit(0);
  }
}

Specifications

  • Version: Java SDK 1.40.0 (also reproduced with 1.38.0)
  • Platform: macOS (Darwin 25.4.0, arm64), OpenJDK 25.0.2
  • Server: Temporal CLI dev server 1.4.1 (server 1.28.0, not affected), 1.6.1 (server 1.30.1) and 1.9.1 (server 1.32.0)
  • Regression: works as expected up to server 1.28.0. Starts with 1.29.0, which includes temporalio/temporal#8238.

Root cause

Since server 1.29.0, a GetWorkflowExecutionHistory long poll only waits until one second before the caller's deadline and then returns an empty response (temporalio/temporal#8238, "GetWorkflowExecutionHistory long poll soft timeout"). Once less than one second is left, the computed wait is zero and the server returns an empty response right away:

On the SDK side, WorkflowClientLongPollHelper.getInstanceCloseEvent uses the same overall deadline for every poll and sends the next request immediately after an empty response. getInstanceCloseEventAsync does the same:

Together, these turn the last second into a loop of "empty response, retry right away" until the deadline. The server change was meant to reduce connection churn, but when the remaining deadline is short it multiplies the number of requests instead.

Possible fix

In the SDK, keep a minimum interval (e.g. 100ms) between polls when the previous poll came back empty, without going past the deadline. Normal long polls, which return after several seconds, are not affected.

Results from a local test with this change applied to the sync path only:

Case Before After
getResult(2s) times out, server 1.32.0 ~4,000 calls 12 calls
Same, server 1.28.0 1 call 1 call
20 concurrent callers 36 of 40 failed with RESOURCE_EXHAUSTED 40 of 40 threw TimeoutException at 2s
Workflow completes during the last second returned right away returned up to 100ms later (15-51ms in our runs)

The async path needs the same rule. Since the SDK still targets Java 8, I'd like to know the preferred way to schedule the delay there (for example, reusing an existing scheduler).

Another option is to change the server so that it doesn't answer immediately when the remaining time is below the buffer. That would cover every SDK, but it's a server change.

Workaround

Don't rely on getResult(timeout) timing out while handling a request. Check the status with describe() first and call getResult only for completed executions, or let the client retry. This is what we ended up doing.

If this direction sounds reasonable, I'm happy to open a PR with the sync and async changes and a unit test. The test uses a fake client that mimics the server's soft timeout; it fails on the current code and passes with the change.

Vorherrschende Sprache
Java
Sterne
433
Forks
257
Ø Merge
2 T. 18 Std.
Gemergte PRs (30 T.)
24

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus temporalio/sdk-java

Alle Issues in temporalio/sdk-java

Ähnliche Issues

Weitere Issues zu Java

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.