Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

E2E Tests: Intermittent TestTerraformModuleTest failures due to stale RBAC permissions from previous test

Abierto
  • ¿Algo no está bien?
#231 6 comentarios 1 reacción 2 asignados Ver en GitHub

@maniSbindra ya está trabajando en esto.

Desde el 26/2/2026.

  • #232 de @copilot-swe-agent — fusionado
  • #237 de @copilot-swe-agent — fusionado

Evaluación

Este issue todavía no se ha evaluado.

Descripción

bug

Summary

The daily scheduled E2E pipeline (🧪 E2E Tests) has a ~27% failure rate (8 failures in the last 30 runs). The most frequent flaky test is TestTerraformModuleTest, which intermittently finds 5 permissions instead of the expected 8.

Root Cause

The tests share a single service principal per (os, type) matrix combination. Each test calls DetachRolesFromSP at startup to remove all role assignments, then waits 15 seconds for Azure RBAC de-propagation before creating a new custom role and starting the MPF iteration loop.

However, 15 seconds is often insufficient for Azure to fully de-propagate the previous test's role assignments. When TestTerraformModuleTest starts after TestTerraformACINoTfvarsFile (which discovers resourcegroups/{read,write,delete}), the SP may still effectively have those permissions cached from the prior test's custom role.

What happens on a failed run

Traced from runs #21550858823 (Jan 31) and #21840777556 (Feb 9):

Iteration What happened Permission discovered
0 azurerm_resource_group.this: Creation complete after 13s — RG created successfully, then workspaces/read denied workspaces/read
1 RG refreshed OK, workspaces/write denied workspaces/write
2 Deployment succeeds → enters destroy phase → "Authorization Successful" workspaces/delete (from destroy)

Result: 5 permissions found (workspaces/{read,write,delete} + deployments/{read,write} from initial)

Expected: 8 permissions (the above + resourcegroups/{read,write,delete})

The 3 missing resourcegroups/* permissions were never surfaced as AuthorizationFailed errors because the SP still had them from the previous test's role assignment that hadn't fully de-propagated.

Key evidence
  • The resource group is created by Terraform (not by MPF — autoCreateResourceGroup is false for all Terraform tests)
  • The RG creation succeeds without any auth error in iteration 0, proving the SP still has resourcegroups/write from the prior test
  • The same 3 permissions are missing in every occurrence (Jan 31 and Feb 9 show identical patterns)
  • The test runs sequentially (-p 1 -parallel 1), so the issue is temporal, not concurrent

Affected Code

  • pkg/usecase/mpfService.go line 112: time.Sleep(15 * time.Second) — wait after role deletion
  • e2eTests/e2eTerraform_test.go:229: assert.Equal(t, 8, len(perms)) — exact count assertion

Failed Runs

Date Run ID Test Error
Feb 24 22370086522 TestARMTemplatMultiResourceTemplateFullDeployment 54 perms, expected ≥57 (same root cause)
Feb 9 21840777556 TestTerraformModuleTest 5 perms, expected 8
Jan 31 21550858823 TestTerraformModuleTest 5 perms, expected 8

Proposed Solutions

  1. Short-term: Increase the RBAC de-propagation wait from 15s to 45s
  2. Medium-term: Replace the fixed sleep with an active probe — poll until a test API call returns 403 AuthorizationFailed, confirming old permissions are fully revoked
  3. Long-term: Use separate service principals per test function to eliminate cross-test RBAC contamination entirely
Lenguaje dominante
Go
Estrellas
66
Forks
11
Merge medio
2 h 7 min
PR fusionados (30 d)
5

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Azure/mpf

Todos los issues de Azure/mpf

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.