Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Observability: Cron enhancements

未关闭
  • 哪里不对？
#1,304 0 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@vprashrex 已经在做这个了。

开始于 2026年10月23日。

评估

这个 Issue 还没有评估数据。

描述

Is your feature request related to a problem?
The current design excludes /cron/* endpoints from tracing, making it difficult to track what a cron run processes since Celery tasks appear as separate root traces. Additionally, failing runs incorrectly report as successful due to the handling of exceptions in process_all_pending_evaluations.

Describe the solution you'd like

  • Trace only when there's actual work, opening spans after polling for pending runs while keeping /cron excluded.
  • Implement error monitoring for runs that end with status: error to correctly report failures in Sentry.
  • Include span attributes like pending run count and processing status for clarity.
  • Ensure consistent timezone settings across monitor configurations.
  • Create tests to validate span existence with pending runs and error reporting for failed runs.
Original issue

Problem

/cron/* endpoints are excluded from tracing on purpose, so an idle cron run doesn't create a trace every minute. The downside is that Celery tasks the cron enqueues (send_eval_completion_notification, fast-eval barriers, iteration resumes) have no parent span and each one shows up as its own root trace. You can't see in one place what a cron run actually processed.

There's a second gap: process_all_pending_evaluations catches every exception and returns {"status": "error"}, so the @sentry_sdk.monitor check-in still reports OK when the run failed.

What we'll do

  • Trace only when there's real work. Keep /cron excluded. Open a span (evaluation.cron.poll, evaluation.cron.poll_stt / poll_tts, plus fast-barrier and iteration-resume dispatch) only after a poll finds pending runs. Tasks enqueued inside nest under it; idle runs produce no trace.
    • crud/evaluations/processing.py → poll_all_pending_evaluations
    • crud/evaluations/cron_utils.py → poll_all_pending_evaluations_by_type
    • crud/evaluations/cron.py → fast-eval barrier and iteration-resume dispatch
  • Report real failures to the cron monitor. When a run ends with status: error, send an error check-in (re-raise, or capture_checkin(..., status=MonitorStatus.ERROR)), so Sentry marks the run failed instead of OK.
  • Span attributes worth having: pending run count, processed / failed / still-processing counts, so a cron trace explains itself.
  • Consistent timezone across monitor configs (evaluations uses UTC, the others Asia/Kolkata).
  • Tests: span exists only when there are pending runs; failed run sends an error check-in.

What Sentry Crons gives us (besides uptime)

  • @monitor / with monitor(...) with monitor_config to create or update the monitor from code (already used in api/routes/cron.py).
  • monitor_config: schedule (crontab or interval), checkin_margin, max_runtime, failure_issue_threshold, recovery_threshold, timezone.
  • Check-in statuses in_progress / ok / error, plus manual capture_checkin(monitor_slug, check_in_id, status) for jobs that can't use the decorator.
  • Missed, failed and timed-out runs become Issues, with per-monitor alerting.
  • Monitor environments: check-ins are tracked per environment (production / staging).
  • CeleryIntegration(monitor_beat_tasks=True, exclude_beat_tasks=[...]) auto-monitors Celery Beat tasks. Not used today because our crons are HTTP-triggered, but an option if they move to Beat.

Out of scope

Uptime monitors, and moving crons to Celery Beat.

主要语言
Python
星标
18
派生
11
平均合并
2 天 2 小时
30 天内合并 PR
33

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

ProjectTech4DevAI/kaapi-backend 的其他 Issue

查看 ProjectTech4DevAI/kaapi-backend 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。