skypilot-org/skypilot

serve: autoscaler `latest_version` resets on controller restart, causing scale churn

开放

#8,562 创建于 2026年1月13日

 (3 条评论) (0 个反应) (0 位负责人)Python (311 个派生)batch import
good first issuegood starter issues

仓库指标

星标
 (4,859 个星标)
PR 合并指标
 (平均合并 2天 23小时) (30 天内合并 146 个 PR)

描述

Summary

  • After Serve controller restarts, autoscaler latest_version stays at INITIAL_VERSION (1) while replicas are launched at the latest service version (e.g., 3).
  • This mismatch causes the autoscaler to repeatedly scale up and then immediately scale down the newly launched replicas, even at 0 RPS.

What we saw

  • One of our service deployments had a constant loop: autoscaler requests 3 scale‑ups (min=2 + overprovision=1), then scales down the same new replicas.
  • The loop starts right after the controller restarts.
  • DB state (example): - services shows current_version=1 but active_versions=[3]. - version_specs only contains version=3.
  • Logs show: - Requests per second 0.0, target replicas computed as 0 or 2, but scale‑ups still requested. - Immediate scale‑down of the replicas that were just launched.

Repro

  1. Launch a service and update it to version > 1.
  2. Restart the controller (or let K8s reschedule it).
  3. Observe autoscaler logs: continuous scale up/down churn, even at 0 RPS.

I can provide a suggestion/fix in a bit.

贡献者指南