Improve the monorepo release process

Ouverte
  • Quelque chose ne va pas ?
#716 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
45/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
github-actions, python
Domaine
ci-cd, release

Piste de recherche

Commencez par lire .github/workflows/pypi-publish.yml ainsi que le fichier associé lambda-layer-publish.yml et les workflows d’images de l’émulateur ECR. Suivez la manière dont les tags de release, les versions des paquets provenant de about.py et les étapes de publication interagissent actuellement, puis comparez le flux de preflight et de vérification proposé au modèle du dépôt JS référencé. La tâche est terminée lorsque les paquets sélectionnés par tag sont publiés en toute sécurité, que les divergences entraînent un abandon avant le téléversement, que les versions déjà publiées sont ignorées et que les releases réussies sont vérifiées.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement pkg:otel pkg:sdk pkg:testing
Problem

Python has no native tooling for releasing multiple packages from one monorepo. There is no equivalent of npm workspaces with per-package publish orchestration. This repository works around the gap with a GitHub Actions workflow, pypi-publish.yml. The workflow builds and publishes every package on every GitHub release. This design has three problems.

1. The release tag does not control what gets published.

The workflow triggers on release: published. It uses the tag only as a checkout ref. No filter connects the tag name to the package matrix. So a release tagged sdk-v2.0.0 builds and uploads all four packages: aws-durable-execution-sdk-python, -otel, -testing, and -insight.

The real selector is each package's __about__.py version. A package publishes when its version is absent from PyPI. A package fails when its version is already there. So the set of published packages depends on which version bumps sit on the tagged commit. It does not depend on what the tag says. A maintainer who intends to release only the SDK can unintentionally publish other packages.

One workflow does read the tag. lambda-layer-publish.yml gates on contains(tag_name, 'otel-v'). That gate forced the combined tag convention sdk-v1.7.0,otel-v0.3.0. The convention packs several package names into one git tag with commas.

2. Already-published versions fail the workflow instead of being skipped.

The publish step does not check PyPI before uploading. It does not set skip-existing. A package with an unchanged version is rebuilt and re-uploaded. PyPI rejects the upload with 400: Uploading new files to releases older than 14 days is not allowed. In practice the -testing package hits this on most releases. So every release since the monorepo conversion has finished with a failed workflow run. This holds even when the intended packages published successfully. A release workflow that is always red trains maintainers to ignore failures. Ignored failures hide real ones.

3. There is no pre-flight plan and no post-publish verification.

Nothing reports which packages will publish before uploads begin. Nothing confirms afterward that PyPI serves the new versions. The first signal of a wrong or missing publish is a user report, or a red job found by reading logs.

How the JS repository avoids this

aws-durable-execution-sdk-js has the same outer shape. A release event triggers a workflow that iterates all publishable packages. The difference is its publish script, iterate-publish-npm.sh. The script runs in two phases.

Phase one is a pre-flight. Before any upload, the script reads the npm registry state for every package. It decides a plan per package: publish under a computed dist-tag, skip because the version is already published, or reject for a policy violation. A reject aborts the run before any upload. An unreadable registry also aborts the run.

Phase two publishes according to the plan. A skip is a deliberate decision and the run stays green. After each upload the script polls the registry until the dist-tag points at the published version.

This makes releases idempotent. A release evaluates six packages and publishes only the ones whose versions moved. Single-package releases such as sdk-2.3.1 and combined releases such as sdk-2.4.0/test-1.1.4/otel-1.0.0 go through the same workflow.

The JS model still does not make the tag authoritative. Package selection there is also version-file-driven. The proposal below goes one step further.

Proposal
  1. Make the tag the source of truth for package selection. Define one tag grammar. For example, sdk-v2.0.0 names a single package and sdk-v2.0.0+otel-v1.0.0 names a combined release. A preflight script parses the tag. Only the named packages are built and published. If a tagged version disagrees with the package's __about__.py, the preflight fails before any upload.
  2. Add a preflight plan job. For each selected package, compare __about__.py against the PyPI JSON API. Emit a plan of publish, skip, or error to the job summary. Downstream jobs consume the plan.
  3. Make publishing idempotent. Set skip-existing: true on pypa/gh-action-pypi-publish. A re-run of a partially failed release then republishes only what is missing.
  4. Verify after publishing. Poll the PyPI JSON API until the new version is served. Bound the retries.
  5. Apply the same tag gate to the downstream release workflows. These are the Lambda layer and ECR emulator image workflows. An SDK-only tag must not rebuild or retag unrelated artifacts.
Acceptance criteria
  1. No tag/publish discrepancy. A release tagged for package X publishes exactly X. Publishing a package not named in the tag is impossible. This holds regardless of which version bumps exist on the tagged commit.
  2. Protection against accidental and inconsistent publishes. The preflight aborts before any upload when a tagged version disagrees with the source version. It also aborts when a policy check fails. Nothing is published on abort.
  3. Successful workflow runs on every legitimate release. An already-published version is a reported skip, not a failure. A green run means everything intended was published and verified. A red run means something went wrong. Re-running a release is safe and idempotent.
Langage dominant
Python
Étoiles
58
Forks
32
Merge moyen
1 j 6 h
PR mergées (30 j)
80

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de aws/aws-durable-execution-sdk-python

Toutes les issues de aws/aws-durable-execution-sdk-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.