perf(cluster): reduce idle replica traffic caused by Commit heartbeats

Abierto
  • ¿Algo no está bien?
#4,707 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@mmmmxa ya está trabajando en esto.

Desde el 2/11/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

api cluster performance
Description

Every partition is a VSR consensus group. Commit signals are sent with a certain frequency in order to communicate the latest commit id and signal the liveness of the primary. When the group is idle, the Commit is sent roughly every 500-700 ms in order to communicate that the primary is still working. It takes up a lot of network bandwidth while the system is doing nothing. This scales with partition size and causes substantial CPU and network bandwidth waste.

Affected area / component

Clustering / replication, Wire protocol / API, Performance

Proposed solution
  1. Batch the messages. Each shard sends heartbeats on a schedule to which every group snaps. This allows putting many commit messages into a single new CommitMessageBatch message, which reduces the traffic significantly.
  2. Heartbeat backoff / adaptive heartbeat. If a heartbeat repeats (has the same view and commit point as the previous one), the group waits longer before the next one, increasing until it hits a time limit set from a config.

Wire change. This adds a #[repr(C)] command and one field to CommitHeader, and bumps REPLICA_WIRE_REVISION from 1 to 2. Please confirm this is acceptable.

Results so far
Image
Alternatives considered
  • Introduce a separate shard-level liveness signal and a separate message to inform backups of Commit information. This requires a protocol change that is harder to reason about and is likely a future improvement.
  • Only batch, with no back-off. Current results show that the batch is responsible for ~83% of traffic reduction and the heartbeat backoff is ~10%. This may seem insignificant, but at scale this reduction is substantial.
Contribution
  • I'm willing to submit a pull request to implement this feature

The issue description is to be updated as the specification for this solidifies.

Lenguaje dominante
Rust
Estrellas
5.2k
Forks
502
Merge medio
3 d 4 h
PR fusionados (30 d)
157

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/iggy

Todos los issues de apache/iggy

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.