Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

No committed wire-format fixtures: internal protocol serialization is only covered by same-version round trips

Aperta
  • Qualcosa non va?
#3,694 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
scala
Ambito
distributed-systems, testing

Direzione di ricerca

Start from SerializationTestKit.scala to see the same-version roundtrip pattern. Review the listed .proto files and SerializerWithStringManifest implementations (e.g., ArteryMessageSerializer.scala) to know what messages need fixtures. Create annotated hex-dump golden files per owning module, pin line endings in .gitattributes, and add CI assertions for byte-identical encode and decode of committed bytes. Done means test-only fixtures exist and both checks run in CI.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Summary

AGENTS.md:41 states "Wire protocol changes must consider rolling upgrade compatibility", but nothing in the build verifies it. Every existing serialization test is a same-version round trip: serialize and deserialize inside one JVM, then compare object equality. There is no test anywhere asserting that bytes produced by one release can be decoded by another, so an accidental wire change is only caught by a production rolling upgrade.

Current coverage

actor-testkit-typed/src/main/scala/org/apache/pekko/actor/testkit/typed/scaladsl/SerializationTestKit.scala:38,50,63:

  def verifySerialization[M](obj: M): M = ...
  def verifySerialization[M](obj: M, assertEquality: Boolean): M = { ... }
  private def roundtrip[M](obj: M): M = { ... }

All 22 *Serialization*Spec.scala files under src/test follow this pattern. A round trip through the same serializer instance cannot detect an encoding change, a renumbered field, a changed manifest string or a reordered collection — it only proves the current code is self-consistent.

The repo contains no committed byte fixtures of any kind:

find . \( -name '*.approved.*' -o -name '*.verified.*' -o -name '*.golden' \) -not -path './target/*' | wc -l
0

The two existing rolling-upgrade tests cover different concerns — cluster/src/test/scala/org/apache/pekko/cluster/JoinConfigCompatCheckerRollingUpdateSpec.scala checks configuration compatibility at join time, and cluster-sharding/src/multi-jvm/scala/org/apache/pekko/cluster/sharding/RollingUpdateShardAllocationSpec.scala checks allocation behaviour during a rolling update. Neither touches encoded bytes.

What is at stake

The internal protocol is carried by 21 .proto files under */src/main/protobuf/:

remote/src/main/protobuf/{ArteryControlFormats,ContainerFormats,SystemMessageFormats,WireFormats}.proto
cluster/src/main/protobuf/ClusterMessages.proto
cluster-typed/src/main/protobuf/{ClusterMessages,ReliableDelivery}.proto
cluster-sharding/src/main/protobuf/ClusterShardingMessages.proto
cluster-sharding-typed/src/main/protobuf/ShardingMessages.proto
cluster-tools/src/main/protobuf/{ClusterClientMessages,DistributedPubSubMessages}.proto
cluster-metrics/src/main/protobuf/ClusterMetricsMessages.proto
distributed-data/src/main/protobuf/{ReplicatedDataMessages,ReplicatorMessages}.proto
persistence/src/main/protobuf/MessageFormats.proto
persistence-query/src/main/protobuf/QueryMessages.proto
persistence-typed/src/main/protobuf/ReplicatedEventSourcing.proto
stream/src/main/protobuf/StreamRefMessages.proto
multi-node-testkit/src/main/protobuf/TestConductorProtocol.proto

plus roughly 21 hand-written SerializerWithStringManifest implementations that carry manifests and serializer identifiers not described by any schema — among them remote/.../serialization/ArteryMessageSerializer.scala, MiscMessageSerializer.scala, DaemonMsgCreateSerializer.scala, cluster/.../protobuf/ClusterMessageSerializer.scala, cluster-sharding/.../protobuf/ClusterShardingMessageSerializer.scala, distributed-data/.../protobuf/{ReplicatedDataSerializer,ReplicatorMessageSerializer}.scala, actor-typed/.../internal/MiscMessageSerializer.scala, stream/.../serialization/StreamRefSerializer.scala.

For the manifest-based ones there is no schema at all: the manifest strings and the identifier values are the contract, and they live in code and in serialization-identifiers entries spread across many reference.conf files. Nothing fails a build if one of them changes.

Proposal

Commit golden byte fixtures for the internal protocol messages and assert two things in CI:

  1. Encode stability — the current code produces byte-identical output for a fixed input.
  2. Decode compatibility — the current code can decode the committed bytes.

(2) is the one that actually protects a rolling upgrade, since it is the direction a mixed-version cluster exercises when an old node talks to a new one.

A readable fixture format matters more than a compact one, because the whole point is that a diff is reviewable in a PR. An annotated hex dump with a small header works well:

# manifest: <manifest string>
# serializer-identifier: <id>
# bytes: <n>
00000000  0a 05 70 65 6b 6b 6f  12 0b 08 c0 a8 01 0a 10 d9  |..pekko.........|
...

so a reviewer can see immediately that field 2 changed from a fixed64 uid to something else, rather than facing an opaque binary diff.

Details worth settling up front:

  • Hash-ordered collections must be snapshotted with a single element. Gossip.members, ORSet dots and similar are emitted in an order that depends on iteration of a hash structure, which is not part of the cross-version contract. A multi-element fixture would fail spuriously.
  • Pin line endings for the fixture directory in .gitattributes, or the fixtures will churn across platforms.
  • Keep the fixtures under the module that owns the message, next to its existing test sources, so they are discovered by the same review.
  • Cover the manifest and the serializer identifier in the header, not just the payload — those are the parts with no schema behind them and are the easiest to change by accident.

A sensible first slice, in decreasing order of traffic on the wire: Artery control messages, system messages, cluster gossip, shard coordinator messages. Those four cover most of what a mixed-version cluster actually exchanges.

Scope

This is test-only. No src/main change, no public API change, no MiMa impact.

Lingua principale
Scala
Stelle
1.7k
Fork
212
Merge medio
21h 9m
PR unite (30g)
98

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/pekko

Tutte le issue di apache/pekko

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.