Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Observability: Cron enhancements

Abierto
  • ¿Algo no está bien?
#1,304 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@vprashrex ya está trabajando en esto.

Desde el 23/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

Is your feature request related to a problem?
The current design excludes /cron/* endpoints from tracing, making it difficult to track what a cron run processes since Celery tasks appear as separate root traces. Additionally, failing runs incorrectly report as successful due to the handling of exceptions in process_all_pending_evaluations.

Describe the solution you'd like

  • Trace only when there's actual work, opening spans after polling for pending runs while keeping /cron excluded.
  • Implement error monitoring for runs that end with status: error to correctly report failures in Sentry.
  • Include span attributes like pending run count and processing status for clarity.
  • Ensure consistent timezone settings across monitor configurations.
  • Create tests to validate span existence with pending runs and error reporting for failed runs.
Original issue

Problem

/cron/* endpoints are excluded from tracing on purpose, so an idle cron run doesn't create a trace every minute. The downside is that Celery tasks the cron enqueues (send_eval_completion_notification, fast-eval barriers, iteration resumes) have no parent span and each one shows up as its own root trace. You can't see in one place what a cron run actually processed.

There's a second gap: process_all_pending_evaluations catches every exception and returns {"status": "error"}, so the @sentry_sdk.monitor check-in still reports OK when the run failed.

What we'll do

  • Trace only when there's real work. Keep /cron excluded. Open a span (evaluation.cron.poll, evaluation.cron.poll_stt / poll_tts, plus fast-barrier and iteration-resume dispatch) only after a poll finds pending runs. Tasks enqueued inside nest under it; idle runs produce no trace.
    • crud/evaluations/processing.py → poll_all_pending_evaluations
    • crud/evaluations/cron_utils.py → poll_all_pending_evaluations_by_type
    • crud/evaluations/cron.py → fast-eval barrier and iteration-resume dispatch
  • Report real failures to the cron monitor. When a run ends with status: error, send an error check-in (re-raise, or capture_checkin(..., status=MonitorStatus.ERROR)), so Sentry marks the run failed instead of OK.
  • Span attributes worth having: pending run count, processed / failed / still-processing counts, so a cron trace explains itself.
  • Consistent timezone across monitor configs (evaluations uses UTC, the others Asia/Kolkata).
  • Tests: span exists only when there are pending runs; failed run sends an error check-in.

What Sentry Crons gives us (besides uptime)

  • @monitor / with monitor(...) with monitor_config to create or update the monitor from code (already used in api/routes/cron.py).
  • monitor_config: schedule (crontab or interval), checkin_margin, max_runtime, failure_issue_threshold, recovery_threshold, timezone.
  • Check-in statuses in_progress / ok / error, plus manual capture_checkin(monitor_slug, check_in_id, status) for jobs that can't use the decorator.
  • Missed, failed and timed-out runs become Issues, with per-monitor alerting.
  • Monitor environments: check-ins are tracked per environment (production / staging).
  • CeleryIntegration(monitor_beat_tasks=True, exclude_beat_tasks=[...]) auto-monitors Celery Beat tasks. Not used today because our crons are HTTP-triggered, but an option if they move to Beat.

Out of scope

Uptime monitors, and moving crons to Celery Beat.

Lenguaje dominante
Python
Estrellas
18
Forks
11
Merge medio
2 d 2 h
PR fusionados (30 d)
33

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ProjectTech4DevAI/kaapi-backend

Todos los issues de ProjectTech4DevAI/kaapi-backend

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.