Align development deployment validation with the reviewed Relay runtime contract

Abierto
#442 15 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
aws, python, shell, terraform

Línea de trabajo

Read _docs/PROCESS.md, _docs/specs/08-aws-development-terraform.md and _docs/runbooks/release-deployments.md, then inspect deploy/task_definitions.py, deploy/update_task_definition_image.py and deploy/deployment_targets.py. Run the relevant TaskDefinitionImageUpdateTests and ci/tests/test_deploy_release_verification.py harnesses first. Done means valid four-secret development definitions pass while invalid references are rejected, Relay configuration is preserved, the ordinary deployment fixture is covered, and the runbook records the handoff.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug infra operations P0

Goal

Make the supported DTC development deployment accept and preserve the reviewed Relay runtime configuration from aws-infra #58 while retaining exact target boundaries. Restore development through the ordinary deployment path after the separately owned infrastructure prerequisites are satisfied. Preserve every course feature and UI.

Evidence and current state

  • Exact source audited: 19e7393489b7158c30bae39f93915a1219f2c757.
  • Authorized reset run 36601136596 completed schema migration, then failed relay_schedule_sync with exit 22. Both website services are stopped. No second database reset is part of this issue.
  • The failed migration:53 definition lacks the Relay URL and secret references. aws-infra #58 already has accepted source providing them; live activation was not performed. Do not duplicate its Terraform implementation.
  • deploy/update_task_definition_image.py:98-99 invokes validate_source_workload(). That calls _secrets() in deploy/task_definitions.py:194-218, which currently permits exactly DATABASE_URL and DJANGO_SECRET_KEY. Correct four-secret definitions from #58 therefore fail before registration.
  • The active updater preserves unrelated environment entries and existing secret references. It does not fetch or synthesize missing runtime credentials.
  • deploy/deploy_website.sh:155-177 reads web/worker source definitions from the services' selected task-definition ARNs; only migration uses the latest family revision. Registering new web/worker family revisions alone does not repair those sources.

Read first and normative ownership

  • _docs/PROCESS.md: separate engineer, tester, PM and on-call gates; no pull requests.
  • _docs/specs/08-aws-development-terraform.md: Relay client boundary, secret confidentiality and separate application/infrastructure permissions. Its opening sandbox physical identifiers are historical; do not revive or widen that retired target.
  • _docs/runbooks/release-deployments.md: supported controller, current targets, exact service-source selection and release gates.
  • deploy/deployment_targets.py: single closed target-definition owner. Current target is website-development, URL https://dev.datatalks.club, namespace website-dev, region eu-west-1, account 387546586013. Production remains independently scoped.
  • _docs/ci/change-selective-ci.md: frozen verification plan and evidence classification.
  • Current coding-standard.md, including the reviewed local version in the shared checkout if it has not landed on main. Read that checkout only; do not copy unrelated work.
  • aws-infra main/dev/README.md, Website Relay runtime contract, main/dev/website_service.tf shared task locals and main/dev/iam.tf execution policy: operator activation ownership.

Scope

  • Give the supported active development promotion gate one authoritative, target-scoped secret-reference contract matching aws-infra #58.
  • Require exactly one each of DATABASE_URL, DJANGO_SECRET_KEY, RELAY_API_KEY, and RELAY_WEBHOOK_SECRET for active development source tasks.
  • Keep existing database/Django reference checks. Relay references must be the development integrations secret in the selected region/account/namespace, with the exact corresponding JSON selector:
    • arn:aws:secretsmanager:eu-west-1:387546586013:secret:website-dev/integrations-<six-character AWS suffix>:RELAY_API_KEY::
    • arn:aws:secretsmanager:eu-west-1:387546586013:secret:website-dev/integrations-<six-character AWS suffix>:RELAY_WEBHOOK_SECRET::
  • Both Relay entries must identify the same integrations secret container. Derive target identifiers through the existing target owner; do not create a second target registry or shell literals.
  • Preserve source RELAY_BASE_URL and both selectors through image promotion for web, worker and migration. Do not guess a private endpoint or rewrite a configured URL. Live URL/configuration correctness remains the infrastructure/runtime prerequisite below.
  • Preserve production's existing exact secret contract and every existing role, family, account, architecture, command, single-container and immutable-image guard.
  • Add a concise recovery handoff to the supported release runbook. No workflow permission expansion is needed.

Non-goals

  • No UI, template, route, feature, notification-policy, package-pin, database-model or schema change.
  • No second reset, migration/Relay bypass, ad hoc secret injection, credential retrieval/logging, Terraform apply, or production mutation under this source issue.
  • No new main/dev apply workflow and no widening of the sandbox Terraform workflow.
  • No broad rewrite or deletion of the retained manual normalizer/release machinery. If a shared helper is extracted, preserve existing callers' distinct contracts; expand changes only for a demonstrated dependency and report it before broadening scope.
  • No progress on frozen answer-check consolidation #439 until development recovery is green.

Dependencies and ownership

Source implementation is ready: aws-infra #58 already supplies the accepted reference schema. Live activation is an external dependency, not a reason to block local implementation or claim recovery from local tests.

  • Engineer: isolated worktree, source/tests/runbook only, frozen uncommitted handoff.
  • Independent tester: verification plan, contract/security boundary and regression evidence.
  • PM: source acceptance after tester pass; independently record outstanding live criteria.
  • Authorized infrastructure operator: existing #58 activation and stopped-service source reconciliation.
  • Orchestrator/on-call: ordinary exact-main deployment after accepted source and operator prerequisites; one CI/deploy observer.

Source acceptance criteria

  • A valid development definition with the four exact references passes the real active module CLI for web, worker and migration; the resulting registration document retains the URL and exact secret entries while updating only the existing owned release fields/command normalization.
  • Missing, duplicate or unexpected secret names; wrong region/account/namespace/container; malformed references; missing, swapped or incorrect JSON selectors; added stage/version selectors; and mismatched Relay container ARNs are refused. No source credential value is printed. Failure occurs before registration of the offending definition, task launch or service promotion; an earlier successfully validated family need not be retrospectively unregistered.
  • Valid production definitions still pass unchanged. Production rejects development Relay references and additional secret names. Existing target, role, family, architecture, container, command and image rejection tests remain effective.
  • The real deploy shell integration fixture models all three corrected development definitions, proves preserved Relay configuration at registration, uses service-selected web/worker revisions, and runs the ordinary migration/schedule/template sequence with reset absent. A rejected source cannot launch migration or promote services. No database reset command is introduced.
  • New tests run in the applicable repository/deployment verification selection. Existing oversized test files are not simply expanded to hide new coverage outside the runner.
  • New handwritten functions are at most 30 lines and files at most 300 lines. Existing oversized modules/functions do not grow; any necessary extraction follows one cohesive validation responsibility and preserves callers. No ternaries or filtered/nested comprehensions are added. Any unavoidable exception is justified explicitly.
  • Engineer and independent tester record exact base/head/worktree state, plan and graph digests, commands/counts and evidence dispositions. Follow the generated verification plan, blocking lint and applicable advisory report; do not narrow a required full gate. PM accepts only after tester pass.
  • The release runbook records the operational handoff below, with source acceptance clearly distinguished from live recovery.

Verification scenarios

  1. Extend coverage at the existing owners: TaskDefinitionImageUpdateTests in core/tests/test_cmp_style_deployment.py provides production and real dev module-CLI examples; ci/tests/test_deploy_release_verification.py provides the real shell/fake-AWS transport harness and service-source selection coverage. Prefer cohesive separate test modules/helpers where size limits require them; update discovery if necessary.
  2. Use inert synthetic ARNs and fake AWS only for local contract tests. At least one authoritative test must fail on the old two-secret validator and pass on the fix. Reuse existing negative guards where they already prove unchanged behavior.
  3. Browser/screenshots: this is deployment control-plane code with no render impact. Product-page screenshot evidence is not_applicable, with that reason in the plan/report. Run the repository-required backend Playwright smoke tier and any additional graph-selected gates; do not invent course visual changes to justify screenshots.
  4. No cloud access, infrastructure mutation or secret-value read is required to complete source acceptance.

Separately owned live recovery prerequisites and evidence

These criteria remain unchecked until their named owner supplies actual evidence. Source acceptance may proceed first; use Refs #442 and leave the issue open while live criteria remain outstanding.

  • [OPERATOR] Complete #58's non-printing secret-shape preflight and review a fresh saved plan from the accepted main/dev revision, using its existing approved operator path. No authorized main/dev apply workflow currently exists; the sandbox workflow must remain scoped to sandbox.
  • [OPERATOR] Reconcile the README restriction limiting activation to task-definition/IAM changes with the necessary web/worker service-pointer updates. The current Terraform resources point those services at its task definitions, but DTC deploy reads their selected revisions. Explicitly review the pointer changes, safe image references and live drift; do not silently bypass the README restriction. Keep both desired counts at zero during preparation. Stop for unrelated state, IAM, networking, database, replacement or service-count changes.
  • [OPERATOR] Provide redacted evidence that all three selected deployment sources have the reviewed Relay URL/selectors and required execution access. Secret content and remote state must not enter issue comments/artifacts. Registering family revisions without fixing stopped service pointers is insufficient.
  • [ON-CALL] After accepted DTC source is merged and exact-SHA CI passes and operator prerequisites are complete, run the ordinary deploy-dev.yml path with confirm_dev_schema_reset EMPTY. Never dispatch another schema reset for this recovery. A push-triggered attempt before operator readiness is not recovery success and must remain failed/blocked.
  • [ON-CALL] Record exact successful run/SHA/image, migration/schedule/template completion, service/runtime health and readiness at https://dev.datatalks.club, then the #58-required deployed sync_relay_schedules --dry-run no-diff and jobs_ingress_selftest OK evidence, without secrets. Failure keeps recovery open; no success record or resumed #439 on partial evidence.

Related: #438, #440, aws-infra #58. #439 remains held.

Lenguaje dominante
Python
Estrellas
0
Forks
0
Merge medio
37 min
PR fusionados (30 d)
36

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de DataTalksClub/website

Todos los issues de DataTalksClub/website

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.