Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Observability: Cron enhancements

Aperta
  • Qualcosa non va?
#1,304 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@vprashrex ci sta già lavorando.

Dal 23/10/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Is your feature request related to a problem?
The current design excludes /cron/* endpoints from tracing, making it difficult to track what a cron run processes since Celery tasks appear as separate root traces. Additionally, failing runs incorrectly report as successful due to the handling of exceptions in process_all_pending_evaluations.

Describe the solution you'd like

  • Trace only when there's actual work, opening spans after polling for pending runs while keeping /cron excluded.
  • Implement error monitoring for runs that end with status: error to correctly report failures in Sentry.
  • Include span attributes like pending run count and processing status for clarity.
  • Ensure consistent timezone settings across monitor configurations.
  • Create tests to validate span existence with pending runs and error reporting for failed runs.
Original issue

Problem

/cron/* endpoints are excluded from tracing on purpose, so an idle cron run doesn't create a trace every minute. The downside is that Celery tasks the cron enqueues (send_eval_completion_notification, fast-eval barriers, iteration resumes) have no parent span and each one shows up as its own root trace. You can't see in one place what a cron run actually processed.

There's a second gap: process_all_pending_evaluations catches every exception and returns {"status": "error"}, so the @sentry_sdk.monitor check-in still reports OK when the run failed.

What we'll do

  • Trace only when there's real work. Keep /cron excluded. Open a span (evaluation.cron.poll, evaluation.cron.poll_stt / poll_tts, plus fast-barrier and iteration-resume dispatch) only after a poll finds pending runs. Tasks enqueued inside nest under it; idle runs produce no trace.
    • crud/evaluations/processing.py → poll_all_pending_evaluations
    • crud/evaluations/cron_utils.py → poll_all_pending_evaluations_by_type
    • crud/evaluations/cron.py → fast-eval barrier and iteration-resume dispatch
  • Report real failures to the cron monitor. When a run ends with status: error, send an error check-in (re-raise, or capture_checkin(..., status=MonitorStatus.ERROR)), so Sentry marks the run failed instead of OK.
  • Span attributes worth having: pending run count, processed / failed / still-processing counts, so a cron trace explains itself.
  • Consistent timezone across monitor configs (evaluations uses UTC, the others Asia/Kolkata).
  • Tests: span exists only when there are pending runs; failed run sends an error check-in.

What Sentry Crons gives us (besides uptime)

  • @monitor / with monitor(...) with monitor_config to create or update the monitor from code (already used in api/routes/cron.py).
  • monitor_config: schedule (crontab or interval), checkin_margin, max_runtime, failure_issue_threshold, recovery_threshold, timezone.
  • Check-in statuses in_progress / ok / error, plus manual capture_checkin(monitor_slug, check_in_id, status) for jobs that can't use the decorator.
  • Missed, failed and timed-out runs become Issues, with per-monitor alerting.
  • Monitor environments: check-ins are tracked per environment (production / staging).
  • CeleryIntegration(monitor_beat_tasks=True, exclude_beat_tasks=[...]) auto-monitors Celery Beat tasks. Not used today because our crons are HTTP-triggered, but an option if they move to Beat.

Out of scope

Uptime monitors, and moving crons to Celery Beat.

Lingua principale
Python
Stelle
18
Fork
11
Merge medio
2g 2h
PR unite (30g)
33

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ProjectTech4DevAI/kaapi-backend

Tutte le issue di ProjectTech4DevAI/kaapi-backend

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.