[Incident] [incident] aoai-model deployment failure: Smoke Azure Codex and Smoke Azure PI engine steps exit 1

Ouverte
  • Quelque chose ne va pas ?
#77,255 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
43/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
azure, github-actions, go
Domaine
ci-cd, infrastructure

Piste de recherche

Start with the linked failed runs and inspect the Execute Codex CLI and Execute Pi CLI step logs for the driver stderr and exit reason. Then check the aoai-model environment variables, federated credential subject, Entra-authenticated AWF proxy, and Azure OpenAI deployment. Done means both smoke workflows pass when rerun with workflow_dispatch.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

deployment-failure incident
Summary

Deployments to the aoai-model environment are failing on main at commit c4fffbf1eb. The failing deployments come from two scheduled smoke workflows, Smoke Azure Codex and Smoke Azure PI. In both, the engine step exits with code 1 about 30 seconds after it starts. The same failure has recurred on every scheduled run since 2026-11-03.

Failing runs and deployments
Workflow Run Failing engine step Agent job
Smoke Azure Codex 43334288414 Execute Codex CLI 131586831744
Smoke Azure PI 43334109551 Execute Pi CLI 131586277653

Failed aoai-model deployments: 7753171242, 7753186837, 7753201642 (all failure, 00:08 to 00:09 UTC on 2026-11-13).

Likely root cause
  • Both engine steps fail the same way. The driver exits with code 1 before a terminal safe output is recorded, and the conclusion job then runs Handle agent failure.
  • Shared setup steps (MCP gateway, safe-outputs config, AWF install, container download) all pass. The failure is limited to the engine step. That points at Azure OpenAI inference through the Entra-authenticated AWF proxy (provider: azure, OIDC via AZURE_TENANT_ID, AZURE_CLIENT_ID, AI_GATEWAY_SCOPE).
  • Earlier auto-filed incidents for these workflows (for example #74268 and #76646) were classified driver_crash, which matches.
  • The failures began 2026-11-03 across six different commit SHAs, so no single recent commit explains them. The firewall bump 838595e127 postdates that first failure and is not the trigger.
  • This is an inference. The engine stderr could not be read (see below).
Evidence limitation

gh run view --log-failed returns HTTP 403 (DIFC policy violation) for these runs, so the engine output was not available. The findings above come from job step metadata, check-run annotations, and prior incident records.

Job step timings and annotations
  • Codex agent job 131586831744: Execute Codex CLI 00:07:38 to 00:08:08 UTC, failure. Annotation: Agent execution exited with code 1.
  • Pi agent job 131586277653: Execute Pi CLI 00:06:39 to 00:07:08 UTC, failure. Annotation: Agent execution exited with code 1.
  • Conclusion jobs 131587683233 and 131587268644: Handle agent failure failed.
  • Sibling smoke-copilot-aoai-entra is intermittent: success on 2026-11-09, failure on 2026-11-11 (run 42929934126).
Suggested remediation
  1. Open the failed Execute Codex CLI and Execute Pi CLI steps in the run links above. Capture the driver stderr and exit reason.
  2. Confirm the aoai-model environment variables AZURE_TENANT_ID, AZURE_CLIENT_ID, and AI_GATEWAY_SCOPE are set. Confirm the federated credential subject matches the environment.
  3. Check token acquisition and health of the Entra-authenticated AWF proxy and the Azure OpenAI deployment used by these smokes.
  4. After a fix, re-run both smoke workflows with workflow_dispatch and confirm they pass before closing this incident.

Generated by 🚨 Deployment Incident Monitor · copilot · routed: haiku55 xhigh · ~3 AIC · ⌖ ~1 AIC · ⊞ 2.9K · ◷

  • expires on Nov 19, 2026, 4:15 PM UTC-08:00
Langage dominant
Go
Étoiles
5.6k
Forks
625
Merge moyen
5 h 42 min
PR mergées (30 j)
796

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de github/gh-aw

Toutes les issues de github/gh-aw

Issues similaires

Plus d'issues Go

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.