Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Memory-bound concurrency for queue consumers

Abierto
  • ¿Algo no está bien?
#1,096 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
aws, azure, go, google-cloud, rabbitmq
Área
distributed-systems, infrastructure

Línea de trabajo

Start by reading the queue-consumer flow, especially where messages are received before waiting for a concurrency slot. Compare the existing GCP Pub/Sub ReceiveSettings.MaxOutstandingMessages setup with its byte flow-control setting; also inspect how SQS, Azure Service Bus, and RabbitMQ handle pulling or prefetching. Done means each queue can enforce the agreed byte limit alongside concurrency, including the oversized-message case, with tests covering those behaviors.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement internalmq

Outpost limits how much work each internal queue consumer holds with a message count: PUBLISH_MAX_CONCURRENCY, DELIVERY_MAX_CONCURRENCY and LOG_MAX_CONCURRENCY. Memory, though, follows count × message size, and nothing bounds the size side.

The same concurrency setting is safe or fatal depending on payload size. A value tuned for throughput with ~1 KB events keeps a few MB in flight. With ~1 MB events it keeps GBs in flight. Each body also exists several times in memory while it's handled: the decoded queue message, the event struct, the destination's request, and the attempt log (the event plus the response).

When it triggers

You need all three:

  • concurrency raised above the default of 1 for throughput
  • large payloads
  • a slow or throttling destination

Slow deliveries keep every slot full. Retries add to new traffic. The process then holds concurrency × payload size × copies of live heap. That memory is live data, so GOMEMLIMIT can't reclaim it, and the process is OOM-killed.

After a restart, the unacked messages are redelivered to a process with every slot free. It fills again and gets killed again, and the loop runs until concurrency is lowered by hand.

Today, avoiding this means lowering concurrency to fit the largest payloads. That caps throughput for everything else. If a deployment's traffic mixes small and large events, no single value is right.

Proposal

Add a byte-based limit per queue next to the existing concurrency setting, for example:

  • PUBLISH_MAX_CONCURRENCY_BYTES
  • DELIVERY_MAX_CONCURRENCY_BYTES
  • LOG_MAX_CONCURRENCY_BYTES

How the two settings work together:

  • Concurrency stays the throughput setting: how many messages are handled at once.
  • Concurrency bytes is the memory setting: how many bytes of messages can be in flight at once, measured by message body size.
  • A message starts only if both limits have room. When either is full, the consumer stops taking messages, so excess load waits in the broker as backlog instead of in process memory.
  • One message larger than the whole byte limit still starts when nothing else is in flight, so an oversized event can't stall the queue.

With a byte limit in place, operators can set concurrency high for throughput without having to size it for their largest payloads. Small events run at full concurrency; large events limit themselves.

Open questions

  • Default. Unset (no behavior change), or derived from GOMEMLIMIT so deployments get a safe bound without tuning?
  • Scope. One limit per queue, matching the existing concurrency settings, or one limit shared across the process, which maps more directly to the memory limit?

Implementation notes

Starting points only; the PR decides the approach.

  • GCP Pub/Sub has native byte flow control (ReceiveSettings.MaxOutstandingBytes). Today we set only MaxOutstandingMessages and leave bytes at the SDK default (1 GB).
  • SQS, Azure Service Bus and RabbitMQ have no native byte limit, so the consumer would enforce it. The consumer currently receives a message before waiting for a concurrency slot. The byte limit should apply before messages are pulled.
  • RabbitMQ subscriptions don't set a prefetch count, so the broker may push more messages than the concurrency setting allows.
Lenguaje dominante
Go
Estrellas
1.1k
Forks
70
Merge medio
2 d 20 h
PR fusionados (30 d)
44

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de hookdeck/outpost

Todos los issues de hookdeck/outpost

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.