test server: retrying activity never times out by ScheduleToClose when its expiration falls on a whole second (`getNanos() != 0` guard in `TestServiceRetryState`)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Read TestServiceRetryState.getBackoffIntervalInSeconds in the Java test server, especially the expiration check described in the issue. Compare the next attempt time with the expiration time even when its nanoseconds are zero, then run the relevant retry-state tests. Done means an activity times out at ScheduleToClose regardless of whether expiration falls on a whole second.
Written by the indexing model from the issue text.
Description
Summary
In the Java test server (temporal-test-server), an activity that keeps failing with a retryable error is supposed to stop retrying once the next attempt would start after its ScheduleToClose deadline. In TestServiceRetryState.getBackoffIntervalInSeconds this check is guarded by expirationTime.getNanos() != 0:
if (expirationTime.getNanos() != 0
&& Timestamps.compare(nextScheduleTime, expirationTime) > 0) {
return new BackoffInterval(RetryState.RETRY_STATE_TIMEOUT);
}
The test server's clock has millisecond precision, so the expiration timestamp has nanos == 0 whenever the activity was scheduled exactly on a whole second (≈1 in 1000 schedules). In that case the check is skipped, and nothing else closes the activity: the ScheduleToClose timer is bound to the attempt number at scheduling time and is discarded as outdated after the first retry. The activity then retries forever (we observed 362 attempts at 20 s intervals well past a 2 h deadline), and the workflow never completes.
The guard looks unnecessary: the constructor already maps "no expiration" to Timestamps.MAX_VALUE, so comparing against it is safe.
Reproduction
Minimal workflow (Kotlin, SDK 1.25.1, TestWorkflowEnvironment with time skipping):
- activity always throws a retryable
ApplicationFailure; ScheduleToCloseTimeout = 2h,RetryOptions(initialInterval = 20m, backoffCoefficient = 1.0), nomaximumAttempts;- workflow catches
ActivityFailureand returns.
Running many independent executions and waiting 5 s (wall clock) for each result:
| test server | executions | never completed | ActivityTaskScheduled.eventTime.nanos of the stuck ones |
|---|---|---|---|
| 1.25.1 | 4514 | 4 | 0 in all 4 |
| 1.40.0 | 409 | 1 | 0 |
1.25.1 with the getNanos() != 0 && part removed |
12000 | 0 (11 schedules had nanos = 0) | — |
A deterministic reproduction is possible by freezing the test server clock on a whole second (we did it with a test-scoped TestServicesStarter variant using Clock.fixed(...)): without the change the activity never closes; with it, it closes by RETRY_STATE_TIMEOUT.
Expected
RETRY_STATE_TIMEOUT whenever the next attempt would be scheduled after the expiration time, regardless of the sub-second part of the expiration timestamp.
Suggested fix
if (Timestamps.compare(nextScheduleTime, expirationTime) > 0) {
return new BackoffInterval(RetryState.RETRY_STATE_TIMEOUT);
}
Versions checked: 1.25.1, 1.26.0, 1.28.0, 1.30.0, 1.34.0, 1.36.0, 1.40.0 — the same condition is present. We currently work around it by shadowing the class in our test classpath.
- Dominant language
- Java
- Stars
- 434
- Forks
- 260
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 24
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from temporalio/sdk-java
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
temporalio/sdk-java#1825 ·
Maintainers usually reply within 1 day
-
Proposal: handle SIGTERM by default to initiate graceful worker shutdownPossibly taken @eamsden claimed this 2 days ago. Open
temporalio/sdk-java#3135 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
temporalio/sdk-java#3132 · 3 comments ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 54/100
temporalio/sdk-java#3125 · 2 comments ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 55/100
temporalio/sdk-java#3124 ·
Maintainers usually reply within 1 day
All issues in temporalio/sdk-java
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
floci-io/floci#5425 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
objectionary/eo-graphs#80 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
objectionary/jucs#141 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
bug good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
repowise-dev/repowise#3335 ·
Maintainers usually reply within 1 day