feat(infra): Preliminary Report: Migrate Classic Application Load Balancers to Global External ALBs

Inizia leggendo infra/frontend/service.tf e infra/backend/service.tf, quindi verifica i servizi backend di staging e le regole di forwarding elencati usando i comandi gcloud describe forniti. Segui le fasi di migrazione di staging documentate prima della produzione, monitora il conteggio delle richieste e le risposte 5xx in Cloud Monitoring e termina impostando gli schemi Terraform e confermando l’assenza di drift con terraform plan nel DevContainer.

Important: This is a preliminary report outlining the roadmap to migrate our existing Classic Application Load Balancers (ALBs) to modern Global External Application Load Balancers (EXTERNAL_MANAGED). With the adoption of Google Provider v8.0.0 in #2807, our load balancers remain on Classic ALB (load_balancing_scheme = "EXTERNAL"). This issue tracks the planned, staged migration to EXTERNAL_MANAGED. An engineer should verify these findings and steps against the live GCP infrastructure before executing any production changes. Useful verification tips and commands are provided at the end of this issue.

---

Overview

In hashicorp/google provider 8.0.0, the default load_balancing_scheme was changed to EXTERNAL_MANAGED (Envoy-based Global External ALB). To avoid disruptive API errors and maintain stability, #2807 explicitly pinned existing load balancers to load_balancing_scheme = "EXTERNAL".

This issue tracks the follow-up project: migrating our frontend and backend load balancers from Classic ALB (EXTERNAL) to Global External ALB (EXTERNAL_MANAGED).

Because Google Cloud disallows changing the load balancing scheme directly in place on active infrastructure via Terraform, this requires an orchestrated, multi-step migration using GCP's migration state machine.

---

Resources Subject to Migration

The following 6 load balancing resources will be migrated across both environments:

Staging Environment (webstatus-dev-public-staging)

• Backend Services (2):
  • staging-frontend-service (infra/frontend/service.tf:L148)
  • staging-backend-service (infra/backend/service.tf:L342)
• Global Forwarding Rules (4):
  • staging-frontend-https-rule (infra/frontend/service.tf:L183)
  • staging-frontend-main-https-rule (infra/frontend/service.tf:L197)
  • staging-backend-https-rule (infra/backend/service.tf:L377)
  • staging-backend-main-https-rule (infra/backend/service.tf:L391)

Production Environment (webstatus-dev-public-prod)

• Backend Services (2):
  • prod-frontend-service
  • prod-backend-service
• Global Forwarding Rules (4):
  • prod-frontend-https-rule
  • prod-frontend-main-https-rule
  • prod-backend-https-rule
  • prod-backend-main-https-rule

---

Step-by-Step Migration Procedure

Execute on Staging first (PROJECT=webstatus-dev-public-staging, ENV=staging), observe for at least one full release cycle, then execute on Production (PROJECT=webstatus-dev-public-prod, ENV=prod).

Phase 1: Migrate Backend Services

Note: In GCP, every backend service attached to a forwarding rule must be migrated to EXTERNAL_MANAGED before migrating any forwarding rules.

1. Set Migration State to PREPARE:
   Prepares Envoy proxy infrastructure alongside existing GFEs.

   gcloud compute backend-services update ${ENV}-frontend-service \
       --project=${PROJECT} --global --external-managed-migration-state=PREPARE
   
   gcloud compute backend-services update ${ENV}-backend-service \
       --project=${PROJECT} --global --external-managed-migration-state=PREPARE
   

   Wait at least 6 minutes for Google edge infrastructure propagation. Verify state reports "Prepared for test".

2. Canary Test via TEST_BY_PERCENTAGE (Recommended for Prod):
   Routes a canary percentage through the modern Envoy data plane.

   gcloud compute backend-services update ${ENV}-frontend-service \
       --project=${PROJECT} --global --external-managed-migration-state=TEST_BY_PERCENTAGE \
       --external-managed-migration-testing-percentage=10
   
   gcloud compute backend-services update ${ENV}-backend-service \
       --project=${PROJECT} --global --external-managed-migration-state=TEST_BY_PERCENTAGE \
       --external-managed-migration-testing-percentage=10
   

   Wait at least 6 minutes. Monitor error rates, latencies, and proxy logs.

3. Transition to TEST_ALL_TRAFFIC:
   Routes 100% of traffic to the Envoy-based ALB infrastructure.

   gcloud compute backend-services update ${ENV}-frontend-service \
       --project=${PROJECT} --global --external-managed-migration-state=TEST_ALL_TRAFFIC
   
   gcloud compute backend-services update ${ENV}-backend-service \
       --project=${PROJECT} --global --external-managed-migration-state=TEST_ALL_TRAFFIC
   

   Wait at least 6 minutes. Verify health checks and traffic stability.

4. Finalize Backend Services to EXTERNAL_MANAGED:

   gcloud compute backend-services update ${ENV}-frontend-service \
       --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED
   
   gcloud compute backend-services update ${ENV}-backend-service \
       --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED
   

   Wait at least 6 minutes.

Phase 2: Migrate Forwarding Rules

Once backend services are EXTERNAL_MANAGED:

gcloud compute forwarding-rules update ${ENV}-frontend-https-rule \
    --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute forwarding-rules update ${ENV}-frontend-main-https-rule \
    --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute forwarding-rules update ${ENV}-backend-https-rule \
    --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute forwarding-rules update ${ENV}-backend-main-https-rule \
    --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

Wait at least 6 minutes.

Phase 3: Rollback Procedure (Safety Mechanism)

GCP maintains a 90-day rollback window:

1. Revert forwarding rules first:
   gcloud compute forwarding-rules update <RULE> --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL
2. Transition backend services back to TEST_ALL_TRAFFIC, wait 6 minutes, then set --load-balancing-scheme=EXTERNAL.

Phase 4: Terraform Code Synchronization

After both environments are migrated:

1. Update infra/frontend/service.tf and infra/backend/service.tf to set load_balancing_scheme = "EXTERNAL_MANAGED" (or remove the explicit setting since it is the default in provider v8).
2. Run terraform plan inside the DevContainer; verify zero infrastructure drift.

---

Tips for Verification & Next Steps

1. Check Live Schemes via gcloud:

   gcloud compute backend-services describe staging-backend-service --project=webstatus-dev-public-staging --global --format="value(loadBalancingScheme)"
   gcloud compute forwarding-rules describe staging-backend-https-rule --project=webstatus-dev-public-staging --global --format="value(loadBalancingScheme)"
   

2. Monitor HTTP Edge Metrics in Cloud Monitoring:
   During TEST_BY_PERCENTAGE and TEST_ALL_TRAFFIC, monitor Cloud Monitoring metric loadbalancing.googleapis.com/https/request_count grouped by response_code_class to verify zero unexpected 5xx responses.
3. Reference Documentation:
   • Google Cloud: Migrate resources from Classic to Global External Application Load Balancer