GKE monitoring component ordering causes persistent Terraform drift

Start with the shared modules/scheduler/gke-cluster module and reproduce the issue using terraform plan -detailed-exitcode after reapplying an unchanged regional GKE cluster. Inspect the Google provider container cluster schema, where enable_components is represented as TypeList, and compare it with the API order shown. Done means an unchanged component set produces an empty plan while additions and removals still appear as changes.

Describe the bug

The shared modules/scheduler/gke-cluster module produces a persistent Terraform diff when GKE returns monitoring_config.enable_components in a different order from the configured list. Reapplying succeeds but does not remove the diff, causing unnecessary cluster updates and failing integration tests that require an empty plan.

Steps to reproduce

1. Deploy a regional GKE Standard cluster using the module's default monitoring_components and enable_dcgm_monitoring settings.
2. Reapply the deployment without changing its configuration.
3. Run terraform plan -detailed-exitcode.

Expected behavior

An unchanged set of monitoring components should produce an empty plan (exit 0), while actual additions and removals remain managed by Terraform.

Actual behavior

The plan exits 2 and proposes an in-place cluster update. The component membership is unchanged, only ordering differs. In the observed API order, STORAGE and HPA precede POD, and DCGM precedes JOBSET. The module defaults put STORAGE/HPA after STATEFULSET and append DCGM after JOBSET.

Version (gcluster --version)

Source build at commit 8efc04f30 (v1.103.0-15-g8efc04f30). Terraform 1.16.1; Google provider 7.46.1; GKE 1.35.8-gke.1036000 in us-central1, REGULAR channel. The same ordering symptom occurred in an earlier run on GKE 1.35.7-gke.1222000.

Blueprint

WIP agentgateway blueprint for https://github.com/GoogleCloudPlatform/cluster-toolkit/issues/6213. It inherits the shared module's monitoring defaults.

Expanded Blueprint

No monitoring overrides: the shared module passes monitoring_components, appending DCGM when enable_dcgm_monitoring is true.

Output and logs

~ monitoring_config {
  ~ enable_components = [
      "SYSTEM_COMPONENTS",
    - "STORAGE",
    - "HPA",
      "POD",
      # DAEMONSET and DEPLOYMENT unchanged
      "STATEFULSET",
    + "STORAGE",
    + "HPA",
      "CADVISOR",
      "KUBELET",
    - "DCGM",
      "JOBSET",
    + "DCGM",
    ]
  }
Plan: 0 to add, 1 to change, 0 to destroy.

Execution environment

• macOS, arm64
• zsh
• Go 1.26.4

Additional context

Related to #6213. The agentgateway deployment and real synchronous/streaming prediction tests pass. This issue is related to reapply stability in the shared GKE module.

A workaround was verified in a fresh live deployment for the agentgateway blueprint being added for #6213: explicitly provide monitoring components in the observed GKE order and disable DCGM for the initial CPU-only example. The broader fix should handle ordering for other blueprints and component selections, including DCGM, without ignoring actual monitoring changes. The 7.46.1 provider schema represents enable_components as schema.TypeList. The provider fix has since been identified; see the adoption status below.

Retain the empty-plan integration check rather than suppressing monitoring changes with ignore_changes.

Provider fix and adoption status (9/18/26)

As identified in this comment, magic-modules#18262 changed the monitoring and logging enable_components fields to sets and shipped in Google provider 8.0.0. This issue now tracks dependency adoption and verification of that fix in Toolkit deployments.

The regenerated agentgateway deployment still selects provider 7.46.1 because transitive modules constrain the Google provider to < 8.0.0. Its root configuration permits versions through 8.2.0, but the effective constraint is the intersection of all module constraints. Compatible module releases must fall within the Toolkit's module version constraints, and existing deployments need an explicit dependency upgrade; provider lock files do not automatically select a newer version.

Keep the blueprint-local component ordering workaround until the dependency upgrade is verified. Keep DCGM disabled independently for the CPU-only agentgateway example. For GPU blueprints, the linked comment describes preserving DCGM by placing it before JOBSET, but that observed ordering is not an API contract.

Completion criteria:

• Resolve compatible module versions and Google provider 8.0.0 or later within Toolkit's supported bounds.
• Verify both a fresh deployment and an upgrade from provider 7.x without the ordering workaround; record resolved module and provider versions.
• Confirm reapplication produces an empty plan, while actual component additions and removals still produce expected changes, including configurations with DCGM.
• Remove temporary ordering overrides where verified, retaining empty-plan integration checks.