Canary deploy times out when autostop stops a Machine between the deploy's two Machine lists

オープン
  • 何か問題がありますか？
#5,326 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@aderk がすでに取り組んでいます。

2026年10月27日 から。

  • #5327 @aderk による — オープン

評価

難易度
3/5
見積もり時間
半日
初心者へのやさしさ
70/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
go
領域
cli

調査の方向性

The reproduction is TestCanaryAcceptsPreservedStoppedUpdateAfterReread; run it against c6de331 to see the health-check timeout. Read internal/command/deploy/plan.go around the skip_launch decision and stopped-outcome acceptance, then machines_deploymachinesapp.go canary phase two to see which Machine list is passed as originalAppState versus oldAppState. Compare with waitForMachine's launchBasisState. Done when the stopped Machine is accepted and the test passes.

索引モデルが issue の本文から書いたものです。

説明

bug

Describe the bug

With --deploy-retries set, a canary deploy timed out at --wait-timeout on Machines that the proxy autostopped during the deploy. flyd left them stopped on update, an outcome PR #5128 accepts, but flyctl compared it against the wrong read of the Machine and waited for health checks instead.

  • Operating system: Linux (CI runner)
  • fly version: observed with 0.4.82, which predates PR #5128. The same sequence still fails at master c6de331 (test below); v0.4.121 has the same code as master here.

** Paste your fly.toml

# Relevant settings only
[deploy]
  strategy = "canary"

[http_service]
  internal_port = 8080
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 1

  [[http_service.checks]]
    path = "/health"
    interval = "30s"
    timeout = "5s"
    grace_period = "60s"

** Command output: **

$ fly deploy --image <image> --wait-timeout 8m --deploy-retries 1
[...]
Updating existing machines in '<app>' with canary strategy
[...]
✔ Machine <canary> is now in a good state
[...]
> Updating machine config for <machine-a>
> Updating <machine-a> [app]
> Updated machine config for <machine-a>
> Waiting for machine <machine-a> to reach a good state
> Machine <machine-a> reached stopped state
> Running smoke checks on machine <machine-a>
> Running machine checks on machine <machine-a>
> Checking health of machine <machine-a>
[...]
Error: failed to update machine <machine-a>: Unrecoverable error: timeout reached waiting for health checks to pass for machine <machine-a>: failed to get VM <machine-a>: Get "https://api.machines.dev/v1/apps/<app>/machines/<machine-a>": net/http: request canceled

(IDs replaced, line indexes removed.)

Sequence (seconds after the first Machine list)

  • 0: before the canary, flyctl lists the Machines (oldAppState). <machine-a> and <machine-b> are started.
  • 52, 65: the proxy autostops them ("has excess capacity, autostopping machine").
  • 87: after the canary, flyctl lists the Machines again (refreshedState). Both are stopped.
  • 88, 152: flyctl updates them with skip_launch: false (what shouldSkipLaunch returns for a Machine started at the first list). flyd logs "machine was in a non-started state prior to the update so leaving the new version stopped" for each.
  • flyctl waits for health checks on the stopped Machine and fails 480 s later, at --wait-timeout.

Expected: flyctl accepts the stopped outcome ("Machine ... was updated and left stopped"), as it does when the autostop lands after the second list.

Actual: flyctl runs smoke, machine and health checks on the stopped Machine and fails at --wait-timeout.

Cause (master c6de331)

A Machine that is started at the first list and stopped at the second is therefore updated with skip_launch: false, and its stopped outcome is rejected. With --deploy-retries 0 both decisions use the same read (machines_deploymachinesapp.go#L1035), so that path is not affected.

Reproduction

TestCanaryAcceptsPreservedStoppedUpdateAfterReread runs updateExistingMachinesWRecovery against a fake Machines API. The first list returns both Machines started; the second returns one stopped; after its update that Machine reads as stopped with an update/stopped/flyd event. At c6de331 the test fails with timeout reached waiting for health checks to pass for machine autostopped-machine. The test is in the linked PR; a tests-only patch that applies to c6de331 is available.

Proposed fix

Judge the stopped outcome against the state that decided skip_launch (originalMachine, or oldMachine when there is none), as waitForMachine already does with launchBasisState.

This report and the patch were prepared with an AI assistant.

主要言語
Go
スター
1.7k
フォーク
320
平均マージ
2日 18時間
マージ済み PR（30日）
72

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

superfly/flyctl のほかの issue

superfly/flyctl の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。