Spanner Outbox/Inbox emulator tests collide on concurrent DDL (need serialized collection)

Open Beginner friendly
#4,224 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
csharp
Domain
testing-qa

Research direction

Start with the Spanner Outbox and Inbox test classes, comparing their setup with the BoxProvisioning tests and their SpannerBoxProvisioning collection. Run the shown dotnet test command with collection parallelisation enabled to reproduce the race, then apply a shared xUnit collection or equivalent serialization and rerun the Spanner filter; the tests should pass concurrently.

Written by the indexing model from the issue text.

Description

Summary

After #4162 (missing EmulatorDetection) is fixed, ~28 Spanner Outbox/Inbox tests fail against the emulator with:

Google.Cloud.Spanner.Data.SpannerException : Operation was rejected because the system is not in a state required for the operation's execution.
---- Grpc.Core.RpcException : Status(StatusCode="FailedPrecondition", Detail="Schema change operation rejected because a concurrent schema change operation or read-write transaction is already in progress.")

The Spanner emulator serialises DDL, but the Outbox (SpannerBinary/SpannerText × Sync/Async) and Inbox test classes run in parallel, and their fixture CREATE/DROP TABLE calls collide on the shared brightertests database. The passing BoxProvisioning tests avoid this via [Collection("SpannerBoxProvisioning")]; the Outbox/Inbox test classes have no such collection.

This was previously masked by the ADC fixture-init failure from #4162 — once EmulatorDetection lets the tests reach the DDL stage, the concurrency limitation surfaces. Different root cause (test parallelism vs emulator DDL serialisation).

Evidence

  • Each affected test passes in isolation.
  • Full ~Spanner filter with collection parallelisation disabled is green:
    SPANNER_EMULATOR_HOST=localhost:9010 GOOGLE_CLOUD_PROJECT=brighter-tests \
      dotnet test tests/Paramore.Brighter.Gcp.Tests/Paramore.Brighter.Gcp.Tests.csproj -f net9.0 \
      --filter "FullyQualifiedName~Spanner" -- xUnit.ParallelizeTestCollections=false
    # 94 passed / 0 failed
    
    Under the default parallel run: ~66 passed / ~28 failed (which tests fail is nondeterministic — it's a race).

Candidate fix (test-only)

Serialise the Spanner Outbox/Inbox test classes:

  • Place them in a shared xUnit collection (mirroring BoxProvisioning's [Collection("SpannerBoxProvisioning")]), or
  • Disable parallelisation for that set.

Context

Surfaced during the /bugfix workflow for #4162. The #4162 fix (production SpannerConnectionProvider/SpannerUnitOfWork + test Const now opt into EmulatorDetection.EmulatorOrProduction) is complete and green; this concurrency issue is tracked separately here.

Dominant language
C#
Stars
2.5k
Forks
297
Avg merge
1d 11h
Merged PRs (30d)
36

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from BrighterCommand/Brighter

All issues in BrighterCommand/Brighter

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.