Cloud Store Backend docs recipe

Use case

Cloud-specific store backends — TupleGCSStoreBackend, TupleS3StoreBackend, and the TupleAzureBlobStoreBackend — were removed in GX 1.0. They were thin wrappers around cloud SDKs and fell outside the library's core scope.

Users upgrading from 0.x / early-1.x (e.g. the 1.4.4 → 1.9.0 report in discussion #11975) hit a wall: their config declares cloud store backends that no longer exist, and there is no documented replacement pattern. They're left guessing how to keep Expectations, Validation Results, Checkpoints, and Data Docs in a bucket. The recommended replacement — an application-level sync workflow — currently lives only in discussion threads and Discourse posts, not in the docs.

We need a documented recipe ("Storing GX artifacts in cloud object storage") that shows the supported sync-based pattern.

Proposed capability

Add a how-to/recipe page to the GX docs covering the supported replacement for the removed cloud store backends. It should be cloud-agnostic in structure with first-class, concrete examples for all three major providers — GCS (gsutil / gcloud storage), S3 (boto3 / AWS CLI), and Azure Blob Storage (azure-storage-blob / az storage blob) — and cover:

1. Sync-based workflow (pull → run → push).

   • Pull the gx/ directory from the bucket into the local environment before running.
   • Run the Checkpoint against local files.
   • Push results back to the bucket under a unique key (suite + run name + timestamp + batch) so concurrent runs don't collide.

2. Config as read-only, single source of truth.

   • Reading configs concurrently is safe — multiple functions can pull the same config from one bucket simultaneously.
   • Writes are last-writer-wins, so config must have a single writer: keep it in Git and push updates from CI, or update from one place manually.
   • Treat config as read-only inside validation functions.

3. Syncing results back safely.

   • Upload only specific result to the validation-results directory, not the whole gx/ folder.
   • Assign a distinct run name per validation, or use the default run name, instead of using a fixed identifier that would overwrite.
   • Avoid destructive syncs (e.g. gsutil rsync -d, aws s3 sync --delete, azcopy sync --delete-destination).

4. Building/publishing Data Docs with a single writer.

   • Don't build Data Docs concurrently from validation functions.
   • Use a separate publishing step — a scheduled job, a post-validation trigger, or a manual workflow — so one writer controls Data Docs generation, built from a single location.

---

Tests

All code shown in the docs must be under test — code snippets are extracted from a real, executed Python script so they can't silently rot. Please follow the repo's existing docs-test pattern rather than pasting untested code blocks.

Where the guide goes

This recipe fits alongside the existing store/Data-Docs configuration guides:

• docs/docusaurus/docs/core/configure_project_settings/configure_metadata_stores/ — for the Expectations / Validation Results / Checkpoints sync workflow.
• docs/docusaurus/docs/core/configure_project_settings/configure_data_docs/ — for the single-writer Data Docs publishing step.

Pattern to copy

configure_data_docs/ is the cleanest reference for a multi-backend guide:

• configure_data_docs/configure_data_docs.md — the guide itself.
• configure_data_docs/_examples/data_docs_local_or_networked.py — the tested example script.
• configure_data_docs/_backends/_local_or_networked.md — per-backend partial (this is the model for giving GCS, S3, and Azure Blob each a first-class variant).

How snippets are wired

Code lives in an _examples/*.py script, wrapped in snippet markers, and the .md references those markers by name — nothing is copy-pasted:

• In the .py:

  # <snippet name=".../_examples/<script>.py - full code example">
  ... code ...
  # </snippet>
  

• In the .md:

  ```python title="Python" name=".../_examples/<script>.py - full code example"
  ```
  

How the test is registered and run

Add an IntegrationTestFixture for each example script in tests/integration/test_script_runner.py (see IntegrationTestFixture in tests/integration/integration_test_fixture.py for the fields):

IntegrationTestFixture(
    name="configure_cloud_store_backend_gcs",
    user_flow_script="docs/docusaurus/docs/core/configure_project_settings/configure_metad
ata_stores/_examples/<script>.py",
    backend_dependencies=[BackendDependencies.GCS],  # BackendDependencies.AWS / .AZURE fo
r the other variants
),

• Cloud flags live in tests/integration/backend_dependencies.py: AWS, GCS, AZURE.
• Run docs tests with pytest -est_script_runner.py, or a single one with `pytest --docs-tests -k "test_docs[configure_cloud_store_backend_gcs]" tests/integration/test_script_runner.py
• See tests/integration/README.md for how the runner sets up isolated environments.

Acceptance

• Cloud store backend recipe is live in docs
• Each backend variant (GCS, S3, Azure Blob) has a tested _examples/*.py script referenced by the guide via snippet tags, and a corresponding IntegrationTestFixture registered with the right backend_dependencies flag.

Is this a net-new capability or an enhancement?

Other / not applicable

Alternatives considered

No response

Additional context

No response