Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

E2E Tests: Intermittent TestTerraformModuleTest failures due to stale RBAC permissions from previous test

Aperta
  • Qualcosa non va?
#231 6 commenti 1 reazione 2 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug

Summary

The daily scheduled E2E pipeline (🧪 E2E Tests) has a ~27% failure rate (8 failures in the last 30 runs). The most frequent flaky test is TestTerraformModuleTest, which intermittently finds 5 permissions instead of the expected 8.

Root Cause

The tests share a single service principal per (os, type) matrix combination. Each test calls DetachRolesFromSP at startup to remove all role assignments, then waits 15 seconds for Azure RBAC de-propagation before creating a new custom role and starting the MPF iteration loop.

However, 15 seconds is often insufficient for Azure to fully de-propagate the previous test's role assignments. When TestTerraformModuleTest starts after TestTerraformACINoTfvarsFile (which discovers resourcegroups/{read,write,delete}), the SP may still effectively have those permissions cached from the prior test's custom role.

What happens on a failed run

Traced from runs #21550858823 (Jan 31) and #21840777556 (Feb 9):

Iteration What happened Permission discovered
0 azurerm_resource_group.this: Creation complete after 13s — RG created successfully, then workspaces/read denied workspaces/read
1 RG refreshed OK, workspaces/write denied workspaces/write
2 Deployment succeeds → enters destroy phase → "Authorization Successful" workspaces/delete (from destroy)

Result: 5 permissions found (workspaces/{read,write,delete} + deployments/{read,write} from initial)

Expected: 8 permissions (the above + resourcegroups/{read,write,delete})

The 3 missing resourcegroups/* permissions were never surfaced as AuthorizationFailed errors because the SP still had them from the previous test's role assignment that hadn't fully de-propagated.

Key evidence
  • The resource group is created by Terraform (not by MPF — autoCreateResourceGroup is false for all Terraform tests)
  • The RG creation succeeds without any auth error in iteration 0, proving the SP still has resourcegroups/write from the prior test
  • The same 3 permissions are missing in every occurrence (Jan 31 and Feb 9 show identical patterns)
  • The test runs sequentially (-p 1 -parallel 1), so the issue is temporal, not concurrent

Affected Code

  • pkg/usecase/mpfService.go line 112: time.Sleep(15 * time.Second) — wait after role deletion
  • e2eTests/e2eTerraform_test.go:229: assert.Equal(t, 8, len(perms)) — exact count assertion

Failed Runs

Date Run ID Test Error
Feb 24 22370086522 TestARMTemplatMultiResourceTemplateFullDeployment 54 perms, expected ≥57 (same root cause)
Feb 9 21840777556 TestTerraformModuleTest 5 perms, expected 8
Jan 31 21550858823 TestTerraformModuleTest 5 perms, expected 8

Proposed Solutions

  1. Short-term: Increase the RBAC de-propagation wait from 15s to 45s
  2. Medium-term: Replace the fixed sleep with an active probe — poll until a test API call returns 403 AuthorizationFailed, confirming old permissions are fully revoked
  3. Long-term: Use separate service principals per test function to eliminate cross-test RBAC contamination entirely
Lingua principale
Go
Stelle
66
Fork
11
Merge medio
2h 7m
PR unite (30g)
5

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Nessun modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Azure/mpf

Tutte le issue di Azure/mpf

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.