E2E Tests: Intermittent TestTerraformModuleTest failures due to stale RBAC permissions from previous test
Nobody has claimed this yet.
Assessment
This issue has not been assessed yet.
Description
Summary
The daily scheduled E2E pipeline (🧪 E2E Tests) has a ~27% failure rate (8 failures in the last 30 runs). The most frequent flaky test is TestTerraformModuleTest, which intermittently finds 5 permissions instead of the expected 8.
Root Cause
The tests share a single service principal per (os, type) matrix combination. Each test calls DetachRolesFromSP at startup to remove all role assignments, then waits 15 seconds for Azure RBAC de-propagation before creating a new custom role and starting the MPF iteration loop.
However, 15 seconds is often insufficient for Azure to fully de-propagate the previous test's role assignments. When TestTerraformModuleTest starts after TestTerraformACINoTfvarsFile (which discovers resourcegroups/{read,write,delete}), the SP may still effectively have those permissions cached from the prior test's custom role.
What happens on a failed run
Traced from runs #21550858823 (Jan 31) and #21840777556 (Feb 9):
| Iteration | What happened | Permission discovered |
|---|---|---|
| 0 | azurerm_resource_group.this: Creation complete after 13s — RG created successfully, then workspaces/read denied |
workspaces/read |
| 1 | RG refreshed OK, workspaces/write denied |
workspaces/write |
| 2 | Deployment succeeds → enters destroy phase → "Authorization Successful" | workspaces/delete (from destroy) |
Result: 5 permissions found (workspaces/{read,write,delete} + deployments/{read,write} from initial)
Expected: 8 permissions (the above + resourcegroups/{read,write,delete})
The 3 missing resourcegroups/* permissions were never surfaced as AuthorizationFailed errors because the SP still had them from the previous test's role assignment that hadn't fully de-propagated.
Key evidence
- The resource group is created by Terraform (not by MPF —
autoCreateResourceGroupisfalsefor all Terraform tests) - The RG creation succeeds without any auth error in iteration 0, proving the SP still has
resourcegroups/writefrom the prior test - The same 3 permissions are missing in every occurrence (Jan 31 and Feb 9 show identical patterns)
- The test runs sequentially (
-p 1 -parallel 1), so the issue is temporal, not concurrent
Affected Code
pkg/usecase/mpfService.goline 112:time.Sleep(15 * time.Second)— wait after role deletione2eTests/e2eTerraform_test.go:229:assert.Equal(t, 8, len(perms))— exact count assertion
Failed Runs
| Date | Run ID | Test | Error |
|---|---|---|---|
| Feb 24 | 22370086522 | TestARMTemplatMultiResourceTemplateFullDeployment |
54 perms, expected ≥57 (same root cause) |
| Feb 9 | 21840777556 | TestTerraformModuleTest |
5 perms, expected 8 |
| Jan 31 | 21550858823 | TestTerraformModuleTest |
5 perms, expected 8 |
Proposed Solutions
- Short-term: Increase the RBAC de-propagation wait from 15s to 45s
- Medium-term: Replace the fixed sleep with an active probe — poll until a test API call returns
403 AuthorizationFailed, confirming old permissions are fully revoked - Long-term: Use separate service principals per test function to eliminate cross-test RBAC contamination entirely
- Dominant language
- Go
- Stars
- 66
- Forks
- 11
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 6
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/mpf
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
CreateUpdateCustomRole returns nil after exhausting its retry budget, reporting success when the role was never updatedPossibly taken A pull request linked to this issue is open or already merged. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Add optional flag which does not destroy the resources created including the custom role definitionOpenenhancement terraform
-
For Terraform azurerm provider resources which use LRO polling add RESOURCE_TYPE/operationStatuses/read permissionsPossibly taken @bgdnext64 claimed this 68 days ago. Openenhancement terraform
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
modelcontextprotocol/go-sdk#1340 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
MHSanaei/3x-ui#6731 · 1 comment ·
Maintainers usually reply within 1 day
-
seccomp: goToNative values for loong64 and mipsle have no matching nativeToSeccomp keyPossibly taken @ricardobranco777 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
native-convergence self-host
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
JakeChampion/lang#11408 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 92/100
open-telemetry/opentelemetry-go-compile-instrumentation#1445 ·
Maintainers usually reply within 2 days