Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Queued log transport should detect (and ideally survive) fork/snapshot-restore without consumer intervention

Abierto
  • ¿Algo no está bien?
#26 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
python
Área
backend, observability

Línea de trabajo

Empieza leyendo destinations/queued.py y luego sigue restart_log_destinations() y restart_observability(), incluido el orden posterior a la restauración en el lado del consumidor descrito en PolicyEngine/policyengine-household-api#1597. Revisa la ruta existente de notificación de drops y la cobertura de pruebas unitarias relevante. Se considera terminado cuando el estado de los procesos obsoletos se notifica con indicaciones para reiniciarlos y un motivo de drop distinto; el self-healing es opcional siempre que siga siendo fail-open y esté libre de condiciones de carrera.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

QueuedLogDestination starts its listener thread at construction and deliberately carries no fork hooks or pid checks (documented in destinations/queued.py). That is the right simplicity trade-off inside the package, but it moves a correctness obligation onto every consumer whose process forks or restores from a memory snapshot: they must know to call restart_observability() at the right lifecycle moment, after credentials are re-materialized.

The failure mode when a consumer misses this is the worst kind: the service serves traffic normally while every Google-bound record silently drops (reason="full" once the queue fills, dead listener thread never drains). Nothing crashes, nothing alerts on the request path.

We just paid this cost in practice: PolicyEngine/policyengine-household-api#1597 adopts 1.4.0 on Modal, whose workers restore from memory snapshots (@modal.enter(snap=True) builds the app — and with it the queued transport — at snapshot creation). The fix was a restart_observability() call in the post-restore hook, plus a unit test pinning the ordering. The next consumer with a forking or snapshotting runtime (gunicorn --preload with fork, celery prefork, another snapshot platform) has to rediscover all of this from scratch.

Ask

Make dead-listener state detectable — and ideally self-healing — inside the package, so consumers are safe by default rather than safe by convention.

Two tiers, smallest first:

  1. Detect and report loudly (minimum). QueuedLogDestination.emit() already counts drops with a throttled report. Add a cheap staleness check — e.g. compare os.getpid() against the pid captured at construction — and when it differs, report through the internal-error channel with an actionable message naming restart_observability() and a distinct drop reason (reason="stale-process" rather than "full"). This turns a silent outage into a diagnosable one for the cost of one integer comparison per emit.

  2. Self-heal (stretch, only if it stays simple). On detecting a stale pid, rebuild the destination the way restart_log_destinations() does. Constraints that killed the previous attempt at this problem space (#23) still apply and are worth restating: fail-open above all, rebuild fresh clients rather than resurrect old ones, no locking or state machine on the hot path, and no rebuild-inside-emit races — if the rebuild can't be made trivially safe (e.g. single os.register_at_fork hook rather than emit-path mutation), tier 1 alone is still a big win.

Related prior art: #22 (accept/emit split), #24 (half-open recovery for disabled destinations, closed as deferred), and the consumer-side integration in PolicyEngine/policyengine-household-api#1597.

🤖 Generated with Claude Code

Lenguaje dominante
Python
Estrellas
0
Forks
0
Merge medio
7 h 49 min
PR fusionados (30 d)
5

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de PolicyEngine/policyengine-observability

Todos los issues de PolicyEngine/policyengine-observability

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.