[Feature Request] stellar CLI for validated, guided deployments of the DDG stages

Start with the merged cli/ scaffold and docs/ddg.md, then review the listed stage inputs, including setIAM.yaml.sample, enableServices.sh, and the fast/stages-aw/ variables. The progress checklist breaks the work into pull requests, but this issue tracks a broad design with open stakeholder questions. Done means completing the remaining CLI phases and guide updates, with tested deployment and teardown flows.

Feature Description

The stellar command line tool, under the cli/ directory, guides an operator through the Detailed Deployment Guide in the docs/ddg.md file from prerequisites to a finished Stage 3. It replaces the manual copy, edit and paste flow and the experimental scripts/deploy.sh and scripts/destroy.sh scripts with one tested tool.

• It validates every prerequisite before anything changes, with an exact fix for each gap.
• It collects the deployment answers once and generates the terraform.tfvars file for every stage under fast/stages-aw/.
• It explains or runs each DDG step in order, with review gates before risky changes and resume after a failure.
• It tears a deployment down safely, scoped to the deployment's prefix.

This issue tracks the design, the progress, and the open questions, so stakeholders can follow and discuss the work in one place.

Use Case

Deploying Stellar Engine today takes roughly 60 manual steps, and several of them fail in ways that only surface partway through Terraform.

• Operators hand-edit terraform.tfvars from the samples, copy provider and *.auto.tfvars.json files out of Cloud Storage, and run a two-pass Stage 0 with a state migration.
• Conditional workarounds live in prose, such as the BigQuery restrictServiceUsage error, external billing grants, and the missing KMS service agent.
• The scripts/deploy.sh script generates tfvars only for Stage 0 and Stage 1, has no unit tests, and exits successfully on failure, as fixed in #303.
• The samples, the guide and the code have drifted apart. For example, the Stage 0 kms_protection_level variable has no default, and the guide lists the wrong Terraform version and script name.
• Stage 0 manages several organization roles authoritatively, so its first apply silently removes existing grants. In a shared organization this can remove access that other teams and platform automation depend on.
• Operators copy the whole repository for each environment and edit it in place, so each copy drifts and misses upstream fixes.
• A deleted project keeps its ID for 30 days, so redeploying an environment with the same prefix fails, as #50 describes.

Proposed Solution

Design principles

• The tool reads facts from the repository instead of copying them. Role, API, group, log sink, custom role and constraint lists come from setIAM.yaml.sample, enableServices.sh and the Stage 0 variables, and sync tests fail when the guide and the code drift apart.
• The terraform.tfvars.sample files stay the single source for defaults such as log sinks, firewall rules and DNS policy rules.
• Checks only read. Changes go through a plan, a summary and an explicit approval.
• Each environment lives in its own directory, so one checkout can manage several deployments, and each records the code version it was deployed from.
• The cli/ project is a self-contained uv project that follows the Google Python Style Guide, with unit tests that use in-memory fakes and never touch the network.

Commands

New environments from existing answers

The stellar init --answers-from <environment> option starts a new environment from another one's answers, so moving from a FedRAMP Moderate test to a DoD IL4 test takes a few keystrokes.

• Every carried value becomes a prompt default, so pressing Enter accepts it and the operator changes only what differs.
• Values that depend on the regime, such as the KMS protection level and the Stage 2 networking flavor, follow the new regime unless the operator set them explicitly.
• A carried value that the new regime rejects, such as an unsupported region, is asked again.
• Run state, logs and the prefix never carry over. The --new-prefix option suggests a fresh prefix, which addresses #50 without random suffixes in Terraform, because .gemini/styleguide.md asks for deterministic prefix tokens.
• Preflight also reports soft-deleted projects that still hold the IDs Stage 0 would create.

Organization IAM safety

• Preflight lists every existing grant that the first Stage 0 apply would remove, role by role.
• Before that apply, stellar deploy snapshots the organization IAM policy and shows a removal review that needs typed confirmation. The operator can skip the gate, and the tool records the skip.
• Terraform can run as a service account while a named user fills the Stage 0 bootstrap_user variable.

Progress

Each pull request is self-contained and reviewable on its own.

• stellar preflight, the CLI scaffold, its CI job, and the guide's instructions to run it, merged in #305
• stellar init and stellar render for Stage 0 and Stage 1, with per-environment directories, --answers-from, and --new-prefix for #50, merged in #357
  • #357 also adds stellar env, stellar imports scan, import blocks for pre-created projects and Assured Workloads workloads, and a CLI-owned Stage 0 phase file that replaces -var bootstrap_user.
  • A lab Stage 0 deployment with these workspaces applied, migrated to the remote state bucket, and plans no changes.
• stellar render for both Stage 2 networking flavors and Stage 3, with CIDR overlap validation
• stellar backup and restore
• stellar guide, stellar deploy and stellar status, with the IAM review gate and known-error remediation
• stellar destroy
• Guide updates that make stellar the primary flow, and deprecation of scripts/deploy.sh and scripts/destroy.sh

Related fixes found while building the tool:

• #303 makes scripts/deploy.sh exit on failure and honor "no" and "skip" answers until the tool replaces it.
• #301 derives the IL5 NGFW landing gateway from the configured region, because IL5 is available in more regions than us-east4.
• #300 matches the fast/stage-links.sh impersonation patterns to the real stage service account names.
• #302 proposes ruff and the Google Python Style Guide repository-wide, tracked in #304.
• #355 lets Stage 0 adopt pre-created projects and an Assured Workloads workload, which #357 uses.
• #356 stops the preflight folder check from counting deleted folders.
• #358 drops the size() guard that IAM rejects from the delegated grant conditions.
• #359 passes the folder number to the folder logging settings.
• #361 lets the Stage 0 folders inherit the organization logging settings under regimes that enforce CMEK for Logging.
• #362 fixes a flaky generate_id test.

Compliance & Deployment Context

• Target Deployment Type(s):
  • US Region Restricted (e.g., Access Policy constraint)
  • FedRAMP Medium
  • FedRAMP High
  • FedRAMP Moderate
  • DoD IL4
  • DoD IL5
  • All / General
• Relevant NIST 800-53r5 Controls: CM-2 and CM-6, because the tool generates a consistent baseline configuration for every stage. AC-2 and AC-6, because preflight and the review gate surface organization IAM changes before Stage 0 applies them.

Reusability Check

Stellar Engine prioritizes reusability.

• I have checked if this functionality can be achieved by extending an existing module or blueprint.
• I have verified that this does not duplicate existing functionality.

The tool reuses the existing stages, samples and helper data unchanged. It replaces scripts/deploy.sh rather than adding a second deployer next to it.

Alternatives Considered

• Extending scripts/deploy.sh keeps the flow in Bash, which makes tfvars generation for nested stage inputs and unit testing hard.
• A standard-library-only Python script avoids dependencies but would parse gcloud text output instead of calling the Google Cloud client libraries.
• Terragrunt could manage per-environment configuration and pinned versions. Its evaluation is part of the future work below, because it overlaps with how FAST already passes outputs between stages through Cloud Storage.

Additional Context

This issue extends #49, which asked to automate the UPDATE_ME substitution in backend.tf files. That placeholder no longer exists, because stages now download generated provider files, and stellar deploy covers the remaining Stage 0 automation.

Open questions for stakeholders:

• Should Stage 0 offer an opt-in additive IAM mode for shared or lab organizations, while production keeps the authoritative default?
• Should a later phase support one canonical, unmodified checkout that deploys several pinned environments and upgrades them in place, and should that use the tool's own environment directories or Terragrunt?
• Is a fully non-interactive stellar deploy --yes mode for repeatable test runs wanted?