Canary deploy times out when autostop stops a Machine between the deploy's two Machine lists

Abierto
  • ¿Algo no está bien?
#5,326 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@aderk ya está trabajando en esto.

Desde el 27/10/2026.

  • #5327 de @aderk — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
Medio día
Aptitud para principiantes
70/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
go
Área
cli

Línea de trabajo

The reproduction is TestCanaryAcceptsPreservedStoppedUpdateAfterReread; run it against c6de331 to see the health-check timeout. Read internal/command/deploy/plan.go around the skip_launch decision and stopped-outcome acceptance, then machines_deploymachinesapp.go canary phase two to see which Machine list is passed as originalAppState versus oldAppState. Compare with waitForMachine's launchBasisState. Done when the stopped Machine is accepted and the test passes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

Describe the bug

With --deploy-retries set, a canary deploy timed out at --wait-timeout on Machines that the proxy autostopped during the deploy. flyd left them stopped on update, an outcome PR #5128 accepts, but flyctl compared it against the wrong read of the Machine and waited for health checks instead.

  • Operating system: Linux (CI runner)
  • fly version: observed with 0.4.82, which predates PR #5128. The same sequence still fails at master c6de331 (test below); v0.4.121 has the same code as master here.

** Paste your fly.toml

# Relevant settings only
[deploy]
  strategy = "canary"

[http_service]
  internal_port = 8080
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 1

  [[http_service.checks]]
    path = "/health"
    interval = "30s"
    timeout = "5s"
    grace_period = "60s"

** Command output: **

$ fly deploy --image <image> --wait-timeout 8m --deploy-retries 1
[...]
Updating existing machines in '<app>' with canary strategy
[...]
✔ Machine <canary> is now in a good state
[...]
> Updating machine config for <machine-a>
> Updating <machine-a> [app]
> Updated machine config for <machine-a>
> Waiting for machine <machine-a> to reach a good state
> Machine <machine-a> reached stopped state
> Running smoke checks on machine <machine-a>
> Running machine checks on machine <machine-a>
> Checking health of machine <machine-a>
[...]
Error: failed to update machine <machine-a>: Unrecoverable error: timeout reached waiting for health checks to pass for machine <machine-a>: failed to get VM <machine-a>: Get "https://api.machines.dev/v1/apps/<app>/machines/<machine-a>": net/http: request canceled

(IDs replaced, line indexes removed.)

Sequence (seconds after the first Machine list)

  • 0: before the canary, flyctl lists the Machines (oldAppState). <machine-a> and <machine-b> are started.
  • 52, 65: the proxy autostops them ("has excess capacity, autostopping machine").
  • 87: after the canary, flyctl lists the Machines again (refreshedState). Both are stopped.
  • 88, 152: flyctl updates them with skip_launch: false (what shouldSkipLaunch returns for a Machine started at the first list). flyd logs "machine was in a non-started state prior to the update so leaving the new version stopped" for each.
  • flyctl waits for health checks on the stopped Machine and fails 480 s later, at --wait-timeout.

Expected: flyctl accepts the stopped outcome ("Machine ... was updated and left stopped"), as it does when the autostop lands after the second list.

Actual: flyctl runs smoke, machine and health checks on the stopped Machine and fails at --wait-timeout.

Cause (master c6de331)

A Machine that is started at the first list and stopped at the second is therefore updated with skip_launch: false, and its stopped outcome is rejected. With --deploy-retries 0 both decisions use the same read (machines_deploymachinesapp.go#L1035), so that path is not affected.

Reproduction

TestCanaryAcceptsPreservedStoppedUpdateAfterReread runs updateExistingMachinesWRecovery against a fake Machines API. The first list returns both Machines started; the second returns one stopped; after its update that Machine reads as stopped with an update/stopped/flyd event. At c6de331 the test fails with timeout reached waiting for health checks to pass for machine autostopped-machine. The test is in the linked PR; a tests-only patch that applies to c6de331 is available.

Proposed fix

Judge the stopped outcome against the state that decided skip_launch (originalMachine, or oldMachine when there is none), as waitForMachine already does with launchBasisState.

This report and the patch were prepared with an AI assistant.

Lenguaje dominante
Go
Estrellas
1.7k
Forks
320
Merge medio
2 d 18 h
PR fusionados (30 d)
72

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de superfly/flyctl

Todos los issues de superfly/flyctl

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.