[bug] azure spot: no fallback to next candidate when VM size has no capacity (SkuNotAvailable)

Abierto
#946 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 4 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
azure, go

Línea de trabajo

Start by reading pkg/provider/azure/data/spot.go, especially selectSpotChoice, and trace how its selected VM size and location reach VM creation. The reproduction shows Azure can return capacity errors such as SkuNotAvailable; inspect how those errors surface during stack updates and how candidate choices are ranked. Done means capacity-related failures try the next ranked candidate, with tests covering retry behavior and exhaustion.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Description

On Azure, spot provisioning fails hard when the selected VM size has no capacity in the selected region. mapt picks a single (size, location) pair and, when Azure rejects it with SkuNotAvailable (capacity restriction), the stack update fails and create exits 1 — no other candidate is tried.

Because the selection is based on eviction rates + prices only (pkg/provider/azure/data/spot.go → selectSpotChoice), which change slowly, the same unavailable pair tends to be picked again on subsequent runs.

Reproduction

mapt v0.14.5 (tekton task tkn/infra-azure-rhel.yaml):

mapt azure rhel create \
  --cpus 4 --memory 32 --gpus 0 --nested-virt --arch amd64 --version 10.1 --disk-size 200 \
  --spot --spot-eviction-tolerance lowest --spot-increase-rate 30 \
  --spot-excluded-regions westindia,centralindia,southindia,australiacentral,australiacentral2,australiaeast,australiasoutheast,southafricanorth,southafricawest \
  ...
Full log output
DEBU running 'mapt azure rhel create'
DEBU Creating RHEL Server
DEBU context initialized for mapt0f84237d
DEBU Based on avg prices for instance types [Standard_E4as_v5 Standard_E4as_v6 Standard_E4bs_v5 Standard_E4ns_v6 Standard_E4s_v4 Standard_E4s_v6 Standard_E4s_v5 Standard_F4ams_v7 Standard_F4ams_v6 Standard_FX4ms_v2 Standard_D8as_v6 Standard_D8as_v5 Standard_D8ns_v6 Standard_D8s_v4 Standard_D8s_v6 Standard_D8s_v5 Standard_E8as_v5 Standard_E8as_v6 Standard_E8bs_v5 Standard_E8ns_v6] is az centralus, current avg price is 0.13 and max price is 0.13 with a score of 0-5
DEBU Spot data: {[Standard_E8ns_v6] 0.16500120000000001 centralus  0}
DEBU managing stack stackAzureLinux-mapt-rhel-podman-desktop-877726682
INFO Updating (stackAzureLinux-mapt-rhel-podman-desktop-877726682):
INFO
INFO  +  pulumi:pulumi:Stack mapt-rhel-podman-desktop-877726682-stackAzureLinux-mapt-rhel-podman-desktop-877726682 creating (0s)
DEBU About to create the VM with compute.NewVirtualMachine
INFO  +  tls:index:PrivateKey main-als-privatekey-user creating (0s)
INFO  +  azure-native:resources:ResourceGroup main-als-rg creating (0s)
INFO  +  tls:index:PrivateKey main-als-privatekey-user created (3s)
INFO  +  azure-native:resources:ResourceGroup main-als-rg created (5s)
INFO  +  azure-native:network:NetworkSecurityGroup main-als-sg creating (0s)
INFO  +  azure-native:network:VirtualNetwork main-als-vn creating (0s)
INFO  +  azure-native:network:PublicIPAddress main-als-pip creating (0s)
INFO  +  azure-native:network:NetworkSecurityGroup main-als-sg created (2s)
INFO  +  azure-native:network:PublicIPAddress main-als-pip created (3s)
INFO  +  azure-native:network:VirtualNetwork main-als-vn created (5s)
INFO  +  azure-native:network:Subnet main-als-sn creating (0s)
INFO  +  azure-native:network:Subnet main-als-sn created (4s)
INFO  +  azure-native:network:NetworkInterface main-als-ni creating (0s)
INFO  +  azure-native:network:NetworkInterface main-als-ni created (2s)
INFO  +  azure-native:compute:VirtualMachine main-als-vm creating (0s)
INFO  +  azure-native:compute:VirtualMachine main-als-vm creating (1s) error: Status=409 Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_E8ns_v6' is currently not available in location 'centralus'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details."
INFO  +  azure-native:compute:VirtualMachine main-als-vm **creating failed** error: Status=409 Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_E8ns_v6' is currently not available in location 'centralus'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details."
INFO  +  pulumi:pulumi:Stack mapt-rhel-podman-desktop-877726682-stackAzureLinux-mapt-rhel-podman-desktop-877726682 creating (27s) error: update failed
INFO  +  pulumi:pulumi:Stack mapt-rhel-podman-desktop-877726682-stackAzureLinux-mapt-rhel-podman-desktop-877726682 **creating failed (26s)** 1 error
INFO Diagnostics:
INFO   azure-native:compute:VirtualMachine (main-als-vm):
INFO     error: Status=409 Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_E8ns_v6' is currently not available in location 'centralus'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details."
INFO
INFO   pulumi:pulumi:Stack (mapt-rhel-podman-desktop-877726682-stackAzureLinux-mapt-rhel-podman-desktop-877726682):
INFO     error: update failed
INFO
INFO Outputs:
INFO     main-alsHost          : "<redacted>"
INFO     main-alsUserPrivatekey: [secret]
INFO     main-alsUsername      : "rhqp"
INFO
INFO Resources:
INFO     + 8 created
INFO     2 errored
INFO
INFO Duration: 28s
INFO
ERRO failed to run update: exit status 1
Expected

When the VM creation fails due to capacity (SkuNotAvailable, AllocationFailed, ZonalAllocationFailed, OverconstrainedAllocationRequest, spot OperationNotAllowed max-price), mapt should try the next best candidate from the ranked spot results (other size and/or location) instead of failing.

Possible approaches
  • Keep the ranked list of spot choices (not just the top one) and on capacity errors destroy/re-target the VM resource with the next candidate, up to N attempts.
  • Pre-filter candidates using SKU restrictions (Resource SKUs API, restrictions with reasonCode) and/or Azure Spot Placement Score, so capacity is part of the score (related: #222).
  • Optional flag to fall back to on-demand in a given location if all spot candidates fail.

Related: #242 (spot failing on Azure due to capacity / price), #222.

Lenguaje dominante
Go
Estrellas
25
Forks
41
Merge medio
5 d 3 h
PR fusionados (30 d)
16

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de redhat-developer/mapt

Todos los issues de redhat-developer/mapt

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.