main-cd: two runs of one core+Plugins pair overwrite each other's tags — set 3.0.0-ci.9985 armed a 9984 portal, satellites decline every Plugins bundle

Cerrado
  • ¿Algo no está bien?
#6,180 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
csharp, docker, python, yaml
Área
ci-cd, devops, infrastructure

Línea de trabajo

Start with .github/workflows/main-cd.yml, focusing on the promote and arm phases, then read .github/scripts/resolve-platform.py and the linked issue #6177 for the proposed run-specific identity change. The issue also identifies node-repo-publish-bake.yml as a publication gate. Done means concurrent runs cannot mix portal and tester images, contradictory sets cannot be armed or published, and main/latest cannot move backwards; the issue notes an operator decision remains for the already-mispointed 9985 tag.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug sev:H

Summary

Two main-cd runs that build the same core + Plugins pair both promote it, and the shared identity tags they write (memex-portal-ai:<core7>-p<plugins7>, <core7>, main, latest) end up on whichever run promotes LAST. When the older run promotes last, a set's tester (resolved by its unique version tag) and its portal (resolved through the shared pair tag) come from two different builds. arm then copies the pair tag onto the set's version tag, so memex-portal-ai:<version> names an image that reports a different version. Every satellite consuming the set declines the Plugins bundles on the platform floor.

Reported from MeshWeaver.Manufacturing#84 (comment https://github.com/Systemorph/MeshWeaver.Manufacturing/issues/84#issuecomment-6004721186). The correction below is that the cause is a race between two runs, not a re-promote that reuses an old build under a new number.

Evidence (measured 2026-10-05/06 against ACR and the run logs)

run 9984 (schedule, reconcile lane) run 9985 (workflow_dispatch rebuild, ref lane)
target core d205295, Plugins 29bfaef core d205295, Plugins 29bfaef (tip b0f38f7 had no completed check)
portal built staging-d205295-37297619921 = sha256:0823d97… reports 3.0.0-ci.9984 staging-d205295-37302108498 = sha256:53aa780… reports 3.0.0-ci.9985 (its own identity check passed)
tester built sha256:9ed0081e… (3.0.0-ci.9984) sha256:7ae811d7… (3.0.0-ci.9985)
promote (phase A+B) 11:48:39 → 11:51:21: d205295, d205295-p29bfaef, main, latest → its own images 11:45:43 → 11:48:10: same tags → its own images
Plugins bake tester 9ed0081e + portal 0823d97, floor 9984 tester 7ae811 + portal 53aa780, floor 9985 (consistent)
  • 9984 had been pending in the reconcile slot behind 9983. It started at 11:19:46, 49 s after 9985 started (11:18:57). The two ran fully in parallel; the concurrency comment in main-cd.yml already admits this overlap ("a rebuild dispatch can still overlap a scheduled reconcile").
  • Run 9989's arm armed 3.0.0-ci.9985 via SRC_PORTAL_TAG=d205295-p29bfaef. Today memex-portal-ai:3.0.0-ci.9985 = sha256:0823d97…, and that image's config reads MESHWEAVER_PLATFORM_VERSION=3.0.0-ci.9984.
  • MeshWeaver.Manufacturing run 37309606772 resolved set 9985 as tester 7ae811 (by its version tag) + portal 0823d97 ("images resolved by tag d205295-p29bfaef"). The bake-seed then declined 134 of 149 Plugins types: platform floor 3.0.0-ci.9985 … is NEWER than the running platform 3.0.0-ci.9984. Run 37309505481 failed the same way. The same Manufacturing commit had been green on set 9983.
  • This is not a one-off. Set 9989's pair tag b0f38f7-p26e14ca names 9988's portal (reports 9988), and set 9993's cd953cb-p59a1cea names 9991's (reports 9991). Set 9995 is consistent.
  • memex (control instance Ops/Status/memex, sampled 2026-10-05 22:58Z) runs memex-portal-ai:3.0.0-ci.9985, i.e. the 9984 image, with two ReplicaSets and converged: false. Both portals' /api/version read commit d205295.

Root cause

  • main-cd.yml promote phase A: promote memex-portal-ai "$SHA" "$SHA-p$PLUGINS_SHORT" writes a tag that names a PAIR, not a run. Phase B moves main/latest with no never-backwards check, although phase D has one.
  • main-cd.yml arm: SRC_PORTAL_TAG=$SHA_SEL-p$PLUGINS_SEL. The comment above it ("tags that name THIS set and nothing else") is false once two runs share a pair. Phase C then writes the version tag with no check of what the image reports.
  • .github/scripts/resolve-platform.py resolve_images: an unarmed set's portal is resolved through pair_tag. The tester comes from its unique version tag. So the two halves of one set come from two runs.

Fix

  • Identity by run, not by pair: #6177 (open, another session) moves arm and the resolver to the per-run staging tag staging-<core7>-<run id> and retires the pair tag. That removes the race at its root.
  • Gates, so a contradicting set can never be armed or sealed again (PR to follow): (1) arm asserts on every architecture that the image it is about to arm reports the version it is armed as. (2) node-repo-publish-bake.yml refuses to publish when any bundle stamps a floor above the portal the publication names. (3) promote phase B never moves main/latest backwards.
  • Owed after both land (an operator decision, not done here): the armed tag memex-portal-ai:3.0.0-ci.9985 still names the 9984 image. Even with the staging-tag resolver, a consumer that reads the version tag first resolves set 9985 to the wrong portal. Re-pointing it at sha256:53aa780…, or letting a newer armed set supersede it, is what makes set 9985 consistent.

Not established

  • Whether memex's converged: false and its pods restarting through the evening are caused by the self-updater seeing a reported 9984 under an armed 9985 tag. I did not read the self-updater's log.
  • Whether any satellite other than Manufacturing consumed 9985 inconsistently. Reinsurance, SocialMedia and Crm were green on the same 12:27 dispatch. Education's red has a different cause (install gate, not the floor).
  • New sets have been stuck since ~14:00Z for an unrelated reason: every portal build fails on jobs.plugins-modules.with.modules is absent or empty (#6155). No consistent set newer than 9995 can appear until that is fixed.
Lenguaje dominante
C#
Estrellas
12
Forks
5
Merge medio
5 h 7 min
PR fusionados (30 d)
981

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Systemorph/MeshWeaver

Todos los issues de Systemorph/MeshWeaver

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.