Observability: metrics, logging, and tracing

Start by mapping the sync engine, API, store, subscriptions, node, and configuration entry points, then decide how the metrics, slog logging, and tracing requirements fit together. Verify the Prometheus endpoint, SyncStatus response, TOML configuration, and each listed metric and span; the issue names no files or tests, so test locations must be identified during research.

Summary

Implement observability infrastructure for apex: Prometheus metrics, structured logging, and OpenTelemetry tracing.

Metrics

Instrument all subsystems with OTel metrics exposed via Prometheus endpoint.

Sync engine

• apex_sync_head (gauge) — last synced height
• apex_sync_network_head (gauge) — upstream network head
• apex_sync_lag_seconds (gauge) — time behind network head
• apex_sync_backfill_duration (histogram) — per-batch backfill latency
• apex_sync_errors_total (counter) — sync errors by type
• apex_sync_backfill_progress_pct (gauge) — backfill completion percentage

API

• apex_rpc_request_duration (histogram) — per-method latency
• apex_rpc_request_total (counter) — per-method call count
• apex_rpc_errors_total (counter) — per-method errors

Store

• apex_store_query_duration (histogram) — SQLite query latency
• apex_store_insert_duration (histogram) — insert latency
• apex_store_size_bytes (gauge) — DB file size

Subscriptions

• apex_subscriptions_active (gauge) — active subscription count
• apex_subscription_deliveries (counter) — messages delivered
• apex_subscription_drops (counter) — messages dropped (slow reader)

Node

• apex_build_info (gauge) — with version labels
• apex_uptime_seconds (counter)

Logging

• Use slog (Go stdlib) — no heavy dependencies like ipfs/go-log
• Structured key-value pairs: height, namespace, method, duration, error
• Configurable log level at startup (and ideally at runtime via admin endpoint)

Tracing

• OpenTelemetry spans for sync fetch, store operations, and RPC handlers
• Configurable exporter (stdout for dev, OTLP for production)

API endpoint

Expose a unified SyncStatus() endpoint returning current synced height, network head, sync state (backfilling/streaming), lag, and active subscriptions in one call. celestia-node spreads this across 4 separate modules.

Configuration

[observability]
metrics_address = "0.0.0.0:9090"   # Prometheus endpoint
log_level = "info"                   # debug, info, warn, error
tracing_enabled = false
tracing_endpoint = ""                # OTLP endpoint