Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

getResult(timeout) busy-loops GetWorkflowExecutionHistory in the last second before the timeout (server 1.29+)

Ouverte
  • Quelque chose ne va pas ?
#3,132 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
55/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
java
Domaine
backend-api-design

Piste de recherche

Start with WorkflowClientLongPollHelper.getInstanceCloseEvent and WorkflowClientLongPollAsyncHelper.getInstanceCloseEventAsync, identified in the issue, and trace how empty poll responses are retried against the shared deadline. The issue proposes a minimum interval between empty polls for both paths and a fake-client unit test that mimics the server soft timeout. Done means the test passes, requests are paced without exceeding the deadline, and the sync and async paths are covered.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

When WorkflowStub.getResult(timeout, unit, ...) (or getResultAsync) waits on a workflow that doesn't complete within the timeout, the SDK re-sends GetWorkflowExecutionHistory back to back, with no delay, for the last second before the deadline. Against a local dev server, a single 2s wait that times out sends roughly 3,000-4,000 history requests. The same code sends 1 request against server 1.28.0, so this looks like a regression that started with server 1.29.0.

With several callers waiting at the same time, the burst also hits the namespace rate limit. getResult then fails with RESOURCE_EXHAUSTED instead of throwing TimeoutException, and it returns later than the requested timeout.

Expected Behavior

getResult(2, TimeUnit.SECONDS, ...) on a workflow that is still running should wait with one or two history long polls and throw TimeoutException after 2 seconds. This is what happens against server 1.28.0 (1 request).

Actual Behavior

GetWorkflowExecutionHistory calls for a single getResult(2s) that times out on a running workflow:

Server SDK GetWorkflowExecutionHistory calls
1.28.0 1.40.0 1
1.30.1 1.38.0 ~3,800-4,300
1.32.0 1.40.0 ~2,900-4,000
  • The calls are packed into the last second before the deadline, so the timeout value barely changes the count. 1s, 5s and 25s waits produced about 2,500-4,600 calls each.
  • getResultAsync behaves the same way (~3,800-4,000 calls).

With concurrent waiters (server 1.32.0, N threads each calling getResult(2s) twice on the same running workflow):

Concurrent callers Threw TimeoutException Failed with RESOURCE_EXHAUSTED
3 0 of 6 6 of 6
10 1 of 20 19 of 20
20 4 of 40 36 of 40

The failed calls got a WorkflowServiceException caused by StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded after 2.0-2.9 seconds. In one run, a call failed with service rate limit exceeded instead. Other calls in the same namespace (DescribeWorkflowExecution and a non-waiting GetWorkflowExecutionHistory) were not throttled in these runs.

We also saw this in production on a self-hosted cluster: every short getResult wait that timed out produced a few hundred history requests. It went away after we replaced getResult with describe() and a retry from the client.

Steps to Reproduce the Problem

  1. Start a dev server with Temporal CLI 1.9.1 (server 1.32.0): temporal server start-dev
  2. Run the program below with Java SDK 1.40.0.
  3. It prints something like getResult(2s) timed out after 2004 ms, GetWorkflowExecutionHistory calls: 2981.
  4. Against a dev server from Temporal CLI 1.4.1 (server 1.28.0), the same program prints calls: 1.
Repro (single file, public APIs only) 
import io.grpc.CallOptions;
import io.grpc.Channel;
import io.grpc.ClientCall;
import io.grpc.ClientInterceptor;
import io.grpc.MethodDescriptor;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.client.WorkflowStub;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.worker.WorkerFactory;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import java.time.Duration;
import java.util.Collections;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;

public class GetResultHistoryFloodRepro {

  @WorkflowInterface
  public interface SleepWorkflow {
    @WorkflowMethod
    void run();
  }

  public static class SleepWorkflowImpl implements SleepWorkflow {
    @Override
    public void run() {
      Workflow.sleep(Duration.ofMinutes(5));
    }
  }

  public static void main(String[] args) throws Exception {
    AtomicInteger historyCalls = new AtomicInteger();
    ClientInterceptor countHistoryCalls =
        new ClientInterceptor() {
          @Override
          public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
              MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
            if ("GetWorkflowExecutionHistory".equals(method.getBareMethodName())) {
              historyCalls.incrementAndGet();
            }
            return next.newCall(method, callOptions);
          }
        };

    WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                .setTarget("127.0.0.1:7233")
                .setGrpcClientInterceptors(Collections.singletonList(countHistoryCalls))
                .build());
    WorkflowClient client = WorkflowClient.newInstance(service);
    WorkerFactory factory = WorkerFactory.newInstance(client);
    factory.newWorker("repro").registerWorkflowImplementationTypes(SleepWorkflowImpl.class);
    factory.start();

    SleepWorkflow workflow =
        client.newWorkflowStub(
            SleepWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue("repro")
                .setWorkflowId("repro-" + System.currentTimeMillis())
                .build());
    WorkflowClient.start(workflow::run);
    WorkflowStub stub = WorkflowStub.fromTyped(workflow);
    Thread.sleep(1000);

    historyCalls.set(0);
    long startedAt = System.nanoTime();
    try {
      stub.getResult(2, TimeUnit.SECONDS, Void.class);
    } catch (TimeoutException expected) {
      // The workflow sleeps for 5 minutes, so the wait times out.
    }
    System.out.printf(
        "getResult(2s) timed out after %d ms, GetWorkflowExecutionHistory calls: %d%n",
        (System.nanoTime() - startedAt) / 1_000_000, historyCalls.get());

    stub.terminate("repro done");
    factory.shutdownNow();
    service.shutdownNow();
    System.exit(0);
  }
}

Specifications

  • Version: Java SDK 1.40.0 (also reproduced with 1.38.0)
  • Platform: macOS (Darwin 25.4.0, arm64), OpenJDK 25.0.2
  • Server: Temporal CLI dev server 1.4.1 (server 1.28.0, not affected), 1.6.1 (server 1.30.1) and 1.9.1 (server 1.32.0)
  • Regression: works as expected up to server 1.28.0. Starts with 1.29.0, which includes temporalio/temporal#8238.

Root cause

Since server 1.29.0, a GetWorkflowExecutionHistory long poll only waits until one second before the caller's deadline and then returns an empty response (temporalio/temporal#8238, "GetWorkflowExecutionHistory long poll soft timeout"). Once less than one second is left, the computed wait is zero and the server returns an empty response right away:

On the SDK side, WorkflowClientLongPollHelper.getInstanceCloseEvent uses the same overall deadline for every poll and sends the next request immediately after an empty response. getInstanceCloseEventAsync does the same:

Together, these turn the last second into a loop of "empty response, retry right away" until the deadline. The server change was meant to reduce connection churn, but when the remaining deadline is short it multiplies the number of requests instead.

Possible fix

In the SDK, keep a minimum interval (e.g. 100ms) between polls when the previous poll came back empty, without going past the deadline. Normal long polls, which return after several seconds, are not affected.

Results from a local test with this change applied to the sync path only:

Case Before After
getResult(2s) times out, server 1.32.0 ~4,000 calls 12 calls
Same, server 1.28.0 1 call 1 call
20 concurrent callers 36 of 40 failed with RESOURCE_EXHAUSTED 40 of 40 threw TimeoutException at 2s
Workflow completes during the last second returned right away returned up to 100ms later (15-51ms in our runs)

The async path needs the same rule. Since the SDK still targets Java 8, I'd like to know the preferred way to schedule the delay there (for example, reusing an existing scheduler).

Another option is to change the server so that it doesn't answer immediately when the remaining time is below the buffer. That would cover every SDK, but it's a server change.

Workaround

Don't rely on getResult(timeout) timing out while handling a request. Check the status with describe() first and call getResult only for completed executions, or let the client retry. This is what we ended up doing.

If this direction sounds reasonable, I'm happy to open a PR with the sync and async changes and a unit test. The test uses a fake client that mimics the server's soft timeout; it fails on the current code and passes with the change.

Langage dominant
Java
Étoiles
433
Forks
257
Merge moyen
2 j 18 h
PR mergées (30 j)
24

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de temporalio/sdk-java

Toutes les issues de temporalio/sdk-java

Issues similaires

Plus d'issues Java

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.