[Tests] Vector-set tests wait for replica AOF sync while VSIM is still running
Maintainers usually reply within 2 days
Assessment
This issue has not been assessed yet.
Description
Describe the bug
ClusterVectorSetTests calls WaitForReplicaAofSync while a tight VSIM loop is still running on the replica. The replica publishes its AOF offset only after in-flight VADD replay finishes (WaitForVectorOperationsToComplete). If that replay does not return, the offset stays at the start of the current batch and the sync wait polls until the test timeout. There is no progress check.
On windows-latest / net8.0 / Release, ClusterVectorSetTests(1000).MultipleReplicasWithVectorSetsAsync failed at 180.1s:
[127.0.0.1:18848]: 1,544088 != [127.0.0.1:18852]: 1,535384
WaitForReplicaAofSync → BackOff → Assert.Fail. One replica (18852) was 8704 bytes behind the primary. Both sides were store version 1. On that same job, the (0) fixture of this test passed in 24.8s, and (1000).MultipleReplicasWithVectorSetsAndDeletesAsync passed in 33.2s.
The CI failure text is the one in the description. Stack:
at ClusterTestUtils.BackOff(...) in test/cluster/Garnet.test.cluster/ClusterTestUtils.cs:line 787
at ClusterTestUtils.WaitForReplicaAofSync(...) in test/cluster/Garnet.test.cluster/ClusterTestUtils.cs:line 3387
at ClusterVectorSetTests.MultipleReplicasWithVectorSetsAsync() in test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs:line 654
Steps to reproduce the bug
- Build Debug (
ExceptionInjectionHelperis compiled out of Release). - Park the replica inside
ApplyVectorSetAdd, beforereplicationBlockEventis decremented, so the AOF consumer cannot publish the end offset of the batch. AResetAndWaitinjection at the start of that method does this. - Start one primary and one replica with AOF enabled.
VADDone 75-byteXB8vector withXPREQ8. - Wait until the injection has parked the replay task.
- Call
WaitForReplicaAofSyncwith a test CTS of a few seconds.
The wait fails with an Assert.Fail message containing != and the two offsets. Re-enable the injection so the replay task leaves the park, then call WaitForReplicaAofSync again. The second call succeeds.
The same order, without an injection, is what CI hit: VSIM is cancelled only after WaitForReplicaAofSync in these methods.
ConcurrentVADDReplicatedVSimsAsyncMultipleReplicasWithVectorSetsAsyncMultipleReplicasWithVectorSetsAndDeletesAsync
Expected behavior
Replica VSIM should be stopped and its task awaited after the primary writes finish, and only then should the test wait for AOF sync. A replay that is still inside the current batch should not sit on the sync loop until [CancelAfter] fires. Search hit counts should be the hits collected while the primary write was still running.
Screenshots
No response
Release version
v2.2.1 and current main (77d677099 at the time of the run). The failing test was added in #2032.
IDE
Not an IDE failure. GitHub Actions, dotnet test of Garnet.test.cluster.vectorsets (windows-latest, net8.0, Release). in https://github.com/microsoft/garnet/actions/runs/37767292226
and some other CI's flaky
OS version
Win 11 26H2(26300) and github actions hosted windows-latest
Additional context
Eighty repeats on a local Windows dev box (net8/net10, Debug and Release, 20 each) passed in 11.7–19.0s, so this is not the normal runtime of the (1000) fixture.
Other WaitForReplicaAofSync call sites under test/cluster cancel and await background command loops before sync. A test-only fix is on yuseok-kim/stop-vsim-before-aof-sync (5ede9d27a): cancel VSIM and await those tasks, then sync. Debug / net8.0 ran the three methods, both fixtures, including both withAttributes cases: 8 passed in 70s.
- Dominant language
- C#
- Stars
- 12.1k
- Forks
- 716
- Avg merge
- 2d 21m
- Merged PRs (30d)
- 57
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/garnet
-
GEOADD updating a member's position keeps the member's old TTL, so the new position is silently deletedPossibly taken @Saad-elnagar claimed this 4 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
microsoft/garnet#2265 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Maintainers usually reply within 2 days
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
Maintainers usually reply within 2 days
-
garnet 在启动时加载Aof报错Possibly taken @TedHartMS claimed this 3 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 48/100
microsoft/garnet#2273 · 3 comments · 1 assignee ·
Maintainers usually reply within 2 days
-
BLPOP with a 0 timeout queued in MULTI never replies at EXEC, and leaves the key unusable for every clientPossibly taken @badrishc claimed this 3 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 55/100
microsoft/garnet#2266 · 1 assignee ·
Maintainers usually reply within 2 days
All issues in microsoft/garnet
Similar issues
-
fix(testing): HeadlessTerminal disposes services synchronouslyPossibly taken @zidad claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
RazorConsole/RazorConsole#388 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
starlinghq/starling#209 ·
Maintainers usually reply within 1 day
-
affected/v17 area/frontend type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 63/100
umbraco/Umbraco-CMS#24323 · 1 comment ·
Maintainers usually reply within 2 days
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
dotnet/roslynator#1859 ·
Maintainers usually reply within 1 day