Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Observability: Cron enhancements

Aberta
  • Algo errado?
#1,304 0 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 1 dia

@vprashrex já está trabalhando nisso.

Desde 23/10/2026.

Avaliação

Esta issue ainda não foi avaliada.

Descrição

Is your feature request related to a problem?
The current design excludes /cron/* endpoints from tracing, making it difficult to track what a cron run processes since Celery tasks appear as separate root traces. Additionally, failing runs incorrectly report as successful due to the handling of exceptions in process_all_pending_evaluations.

Describe the solution you'd like

  • Trace only when there's actual work, opening spans after polling for pending runs while keeping /cron excluded.
  • Implement error monitoring for runs that end with status: error to correctly report failures in Sentry.
  • Include span attributes like pending run count and processing status for clarity.
  • Ensure consistent timezone settings across monitor configurations.
  • Create tests to validate span existence with pending runs and error reporting for failed runs.
Original issue

Problem

/cron/* endpoints are excluded from tracing on purpose, so an idle cron run doesn't create a trace every minute. The downside is that Celery tasks the cron enqueues (send_eval_completion_notification, fast-eval barriers, iteration resumes) have no parent span and each one shows up as its own root trace. You can't see in one place what a cron run actually processed.

There's a second gap: process_all_pending_evaluations catches every exception and returns {"status": "error"}, so the @sentry_sdk.monitor check-in still reports OK when the run failed.

What we'll do

  • Trace only when there's real work. Keep /cron excluded. Open a span (evaluation.cron.poll, evaluation.cron.poll_stt / poll_tts, plus fast-barrier and iteration-resume dispatch) only after a poll finds pending runs. Tasks enqueued inside nest under it; idle runs produce no trace.
    • crud/evaluations/processing.py → poll_all_pending_evaluations
    • crud/evaluations/cron_utils.py → poll_all_pending_evaluations_by_type
    • crud/evaluations/cron.py → fast-eval barrier and iteration-resume dispatch
  • Report real failures to the cron monitor. When a run ends with status: error, send an error check-in (re-raise, or capture_checkin(..., status=MonitorStatus.ERROR)), so Sentry marks the run failed instead of OK.
  • Span attributes worth having: pending run count, processed / failed / still-processing counts, so a cron trace explains itself.
  • Consistent timezone across monitor configs (evaluations uses UTC, the others Asia/Kolkata).
  • Tests: span exists only when there are pending runs; failed run sends an error check-in.

What Sentry Crons gives us (besides uptime)

  • @monitor / with monitor(...) with monitor_config to create or update the monitor from code (already used in api/routes/cron.py).
  • monitor_config: schedule (crontab or interval), checkin_margin, max_runtime, failure_issue_threshold, recovery_threshold, timezone.
  • Check-in statuses in_progress / ok / error, plus manual capture_checkin(monitor_slug, check_in_id, status) for jobs that can't use the decorator.
  • Missed, failed and timed-out runs become Issues, with per-monitor alerting.
  • Monitor environments: check-ins are tracked per environment (production / staging).
  • CeleryIntegration(monitor_beat_tasks=True, exclude_beat_tasks=[...]) auto-monitors Celery Beat tasks. Not used today because our crons are HTTP-triggered, but an option if they move to Beat.

Out of scope

Uptime monitors, and moving crons to Celery Beat.

Linguagem predominante
Python
Estrelas
18
Forks
11
Merge médio
2d 2h
PRs com merge (30d)
33

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de ProjectTech4DevAI/kaapi-backend

Todas as issues de ProjectTech4DevAI/kaapi-backend

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.