flyteorg/flyte

[flyte2] Instrument the runs service reconcilers (abort-reconciler) with Prometheus metrics

Aberta

#7.449 aberto em 29 de mai. de 2026

 (2 comentários) (0 reação) (1 responsável)Python (378 forks)batch import
flyte2good first issue

Métricas do repositório

Stars
 (3.705 estrelas)
Métricas de merge de PR
 (Mesclagem média 3d 8h) (114 fundiu PRs em 30d)

Description

Part of #7445. Depends on #7446 (the /metrics endpoint + Scope must exist first).

Summary

Add Prometheus metrics to the runs service background reconcilers (starting with the abort reconciler) to observe queue depth, processing throughput, retries, and failures.

Background

runs/service/abort_reconciler.go runs as a background worker (registered in runs/setup.go via sc.AddWorker("abort-reconciler", ...)). It has a worker pool, a bounded queue (QueueSize: 1000), and retry logic (MaxAttempts, InitialDelay, MaxDelay). None of this is currently observable via metrics.

What to do

  1. Thread the metrics Scope (from #7446) into service.NewAbortReconciler(...) (extend its config/constructor).
  2. Emit metrics such as:
    • current queue depth / pending items (gauge)
    • items processed (counter, labeled by success/failure)
    • retries / attempts (counter)
    • per-item processing latency (timer/histogram)

Acceptance criteria

  • /metrics exposes abort-reconciler queue depth, processed count (success/failure), retry count, and processing latency.
  • Metrics use a dedicated sub-scope, e.g. scope.NewSubScope("abort_reconciler"), created once.
  • A unit test verifies that processing an item updates the relevant counters/gauges.

Pointers

  • runs/service/abort_reconciler.go — the reconciler implementation and its run loop.
  • runs/setup.go:64-73 — where NewAbortReconciler is constructed and registered as a worker.
  • flytestdlib/promutils/scope.goScope helpers (MustNewGauge, MustNewCounter, MustNewStopWatch, NewSubScope).

Notes for contributors

  • The gauge for queue depth should be updated as items are enqueued/dequeued (or sampled periodically).
  • This is independent of #7447 and #7448; all three consume the same Scope from #7446.

Guia do colaborador