Move the Terraform workflows off static AWS credentials onto OIDC

Overview

We need terraform-plan.yaml and terraform-apply.yaml to authenticate to AWS by assuming a role through GitHub's OIDC provider instead of passing a static access key and secret, because devops-security is the repo that builds the org's GitHub Actions OIDC and is the last repo still using long-lived credentials to reach AWS.

Action Items

• Read the current position before changing anything. Both workflows pass aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} and aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} to aws-actions/configure-aws-credentials@v4, and neither declares permissions: id-token: write at the job or workflow level. Line numbers drift — find the step by its name, Configure AWS Credentials.
• Confirm what the secrets actually are. They are the access key of IAM user devops-iam-github-action (key AKIAQQWOSJEPUH74UTOJ, created 2024-02-19, never rotated, still in active use). Verify that nothing outside these two workflows uses that user before planning to delete it — aws iam get-access-key-last-used and CloudTrail on the username are the two places to look.
• Do not create the roles with module "aws-gha-oidc-providers". That module creates an aws_iam_openid_connect_provider, and AWS permits exactly one provider per URL per account. arn:aws:iam::035866691871:oidc-provider/token.actions.githubusercontent.com already exists — the incubator instantiation created it — so a second instantiation fails with EntityAlreadyExists. The module is the right shape to copy and the wrong thing to call.
• Design two roles, not one. The plan/apply privilege split is the part of the pattern that static keys have no equivalent for, and it is a security win here alongside retiring the keys.
  • devops-security-tf-plan — read-only, sub scoped to repo:hackforla/devops-security:ref:refs/heads/* and repo:hackforla/devops-security:pull_request.
  • devops-security-tf-apply — arn:aws:iam::aws:policy/AdministratorAccess, sub scoped to repo:hackforla/devops-security:ref:refs/heads/main only. The branch scoping is what keeps this safe: a branch or pull request cannot assume this role at all.
  • Both trust arn:aws:iam::035866691871:oidc-provider/token.actions.githubusercontent.com with token.actions.githubusercontent.com:aud = sts.amazonaws.com.
  • AdministratorAccess on the apply role is a decision, not a derivation, and is worth recording as one: the IAM user CI runs as today has a narrower set — managed TerraformIAM, TerraformDynamoDBAccess, DevopsSecurityTerraformBucketAccess, AmazonS3ReadOnlyAccess, plus inline GitHubActionsOIDCThumbprintUpdatePolicy — so this widens the apply path rather than matching it one-for-one. It matches incubator-tf-apply, which is the same shape one repo over. When hackforla/incubator#138 reports back on narrowing incubator's apply privileges, its conclusions apply here too.
• Work out what the plan role needs beyond ReadOnlyAccess before creating it, because a plan is not a read-only operation against this backend. terraform/prod.backend.tfvars sets dynamodb_table = "hfla_ops_terraform_table", so every plan acquires and releases a state lock, which needs dynamodb:PutItem and dynamodb:DeleteItem — neither of which ReadOnlyAccess grants. The existing TerraformDynamoDBAccess policy grants exactly this set and can be attached to the plan role. Note incubator-tf-plan carries only ReadOnlyAccess plus a secrets-read policy and appears to lack lock-write permission entirely, so check how incubator's plan actually behaves rather than copying it — either it runs with locking disabled or something else is going on, and the answer decides whether this role needs the DynamoDB policy or the workflow needs -lock=false.
• Settle the two tagging questions before creating the roles, and record the answers on this issue. A tag is hard to change once an audit depends on it.
  • The exact tag key and value marking these roles as deliberately outside Terraform. No immune or exempt convention exists in any of the three repos today. The only tagging precedent is user_tags on IAM users in terraform/aws-users.tf, which uses quoted title-case keys ("Project", "Access Level") — match that shape or diverge on purpose.
  • Which audit the tag exempts the roles from, by name. Decision record "Use it or lose it policy for IAM User Accounts" is about IAM users and these are roles, so the tag may be pre-empting an audit that does not exist. If it cannot be named, the tag documents nothing.
• Create the two roles by hand in account 035866691871, and capture each role's trust policy, attached policies and tags before and after the change. This is an AWS write with no PR trail, so it follows the same discipline as the org-account work: one change at a time, verified after each. These roles are created manually on purpose — it dissolves the bootstrap problem where the Terraform that creates the role is run by the workflow that needs it — which is what makes the switch below a single PR.
• Write down, in terraform/aws-gha-oidc-providers.tf itself, that these two roles exist outside Terraform and why. A reader of that file will otherwise see incubator's roles declared and devops-security's absent and "fix" it, which re-creates the bootstrap problem this decision removed. The note belongs where they will be standing, not only in this issue.
• Switch both workflows in one PR. Replace the two secrets.AWS_* inputs with role-to-assume / role-session-name / aws-region: us-west-2, and add permissions: id-token: write alongside the existing contents: read. hackforla/incubator's .github/workflows/terraform-plan.yaml is the working example to copy.
• Verification does not happen by itself here, and this is the trap. Both workflows filter on paths: ['**/*.tf'], so a PR that changes only workflow files triggers neither one — the switch would sit unexercised until someone else's Terraform change, possibly weeks later. Include a no-op .tf change in the same PR (a comment line is enough) so the plan runs on the PR and the apply runs on merge.
• After the PR merges, confirm the apply run assumed devops-security-tf-apply and succeeded, and that the plan comment appeared on the PR before merge. Both are observable only after the fact, which is why they are listed here rather than as branch checks.
• After both runs are confirmed green, retire the credentials: delete access key AKIAQQWOSJEPUH74UTOJ, delete the repository secrets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and decide whether the IAM user devops-iam-github-action itself should go. Note the user is not currently declared in Terraform — verified after hackforla/devops-security#180 merged — so deleting it also removes an unmanaged resource; if it is kept instead, it needs to be brought into terraform/aws-users.tf. Do not do any of this before the runs are green — it is the rollback path.

Resources/Instructions

• .github/workflows/terraform-plan.yaml and .github/workflows/terraform-apply.yaml — the two files that change. The credential step is identical in both.
• terraform/aws-gha-oidc-providers.tf — where the incubator roles are declared, and where the note about these roles living outside Terraform belongs.
• terraform/modules/aws-gha-oidc-providers/ — the module to read for the trust-policy shape. Do not instantiate it; see the action item above.
• terraform/prod.backend.tfvars — the S3 backend and the DynamoDB lock table the plan role has to be able to write to.
• hackforla/incubator .github/workflows/terraform-plan.yaml and terraform-apply.yaml — the working OIDC example, assuming arn:aws:iam::035866691871:role/incubator-tf-plan.
• hackforla/devops-security#170 — bumps aws-actions/configure-aws-credentials v4→v6 and dflook/terraform-* v1→v3 in these same two files, and rewrites the very step this issue rewrites. Whichever starts first, check the other: if this issue lands first, #170 should close itself as covered for the configure-aws-credentials half.
• hackforla/devops-security#187 — stops the merge-triggered apply auto-approving itself, and adds a workflow_dispatch trigger to terraform-apply.yaml. It is blocked on this issue: that dispatch path is a deliberately unreviewed apply, and it is only safe once devops-security-tf-apply scopes sub to refs/heads/main, because the static keys it replaces carry no branch scoping at all. #187 also hits the same paths: ['**/*.tf'] verification trap described above. Nothing in this issue needs to change for it — but if the apply role here ends up not branch-scoped, #187 needs revisiting before it is worked.
• hackforla/incubator#139 (Terraform plan via Identity Center) and hackforla/incubator#138 (restrict incubator apply privileges) — separate work with the same goal of retiring long-lived AWS credentials. #138 is researching the plan/apply privilege split this issue applies.
• AWS account 035866691871, region us-west-2.