main-cd: two runs of one core+Plugins pair overwrite each other's tags — set 3.0.0-ci.9985 armed a 9984 portal, satellites decline every Plugins bundle

Đã đóng
  • Có gì không ổn?
#6,180 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
25/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
csharp, docker, python, yaml
Lĩnh vực
ci-cd, devops, infrastructure

Hướng nghiên cứu

Start with .github/workflows/main-cd.yml, focusing on the promote and arm phases, then read .github/scripts/resolve-platform.py and the linked issue #6177 for the proposed run-specific identity change. The issue also identifies node-repo-publish-bake.yml as a publication gate. Done means concurrent runs cannot mix portal and tester images, contradictory sets cannot be armed or published, and main/latest cannot move backwards; the issue notes an operator decision remains for the already-mispointed 9985 tag.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug sev:H

Summary

Two main-cd runs that build the same core + Plugins pair both promote it, and the shared identity tags they write (memex-portal-ai:<core7>-p<plugins7>, <core7>, main, latest) end up on whichever run promotes LAST. When the older run promotes last, a set's tester (resolved by its unique version tag) and its portal (resolved through the shared pair tag) come from two different builds. arm then copies the pair tag onto the set's version tag, so memex-portal-ai:<version> names an image that reports a different version. Every satellite consuming the set declines the Plugins bundles on the platform floor.

Reported from MeshWeaver.Manufacturing#84 (comment https://github.com/Systemorph/MeshWeaver.Manufacturing/issues/84#issuecomment-6004721186). The correction below is that the cause is a race between two runs, not a re-promote that reuses an old build under a new number.

Evidence (measured 2026-10-05/06 against ACR and the run logs)

run 9984 (schedule, reconcile lane) run 9985 (workflow_dispatch rebuild, ref lane)
target core d205295, Plugins 29bfaef core d205295, Plugins 29bfaef (tip b0f38f7 had no completed check)
portal built staging-d205295-37297619921 = sha256:0823d97… reports 3.0.0-ci.9984 staging-d205295-37302108498 = sha256:53aa780… reports 3.0.0-ci.9985 (its own identity check passed)
tester built sha256:9ed0081e… (3.0.0-ci.9984) sha256:7ae811d7… (3.0.0-ci.9985)
promote (phase A+B) 11:48:39 → 11:51:21: d205295, d205295-p29bfaef, main, latest → its own images 11:45:43 → 11:48:10: same tags → its own images
Plugins bake tester 9ed0081e + portal 0823d97, floor 9984 tester 7ae811 + portal 53aa780, floor 9985 (consistent)
  • 9984 had been pending in the reconcile slot behind 9983. It started at 11:19:46, 49 s after 9985 started (11:18:57). The two ran fully in parallel; the concurrency comment in main-cd.yml already admits this overlap ("a rebuild dispatch can still overlap a scheduled reconcile").
  • Run 9989's arm armed 3.0.0-ci.9985 via SRC_PORTAL_TAG=d205295-p29bfaef. Today memex-portal-ai:3.0.0-ci.9985 = sha256:0823d97…, and that image's config reads MESHWEAVER_PLATFORM_VERSION=3.0.0-ci.9984.
  • MeshWeaver.Manufacturing run 37309606772 resolved set 9985 as tester 7ae811 (by its version tag) + portal 0823d97 ("images resolved by tag d205295-p29bfaef"). The bake-seed then declined 134 of 149 Plugins types: platform floor 3.0.0-ci.9985 … is NEWER than the running platform 3.0.0-ci.9984. Run 37309505481 failed the same way. The same Manufacturing commit had been green on set 9983.
  • This is not a one-off. Set 9989's pair tag b0f38f7-p26e14ca names 9988's portal (reports 9988), and set 9993's cd953cb-p59a1cea names 9991's (reports 9991). Set 9995 is consistent.
  • memex (control instance Ops/Status/memex, sampled 2026-10-05 22:58Z) runs memex-portal-ai:3.0.0-ci.9985, i.e. the 9984 image, with two ReplicaSets and converged: false. Both portals' /api/version read commit d205295.

Root cause

  • main-cd.yml promote phase A: promote memex-portal-ai "$SHA" "$SHA-p$PLUGINS_SHORT" writes a tag that names a PAIR, not a run. Phase B moves main/latest with no never-backwards check, although phase D has one.
  • main-cd.yml arm: SRC_PORTAL_TAG=$SHA_SEL-p$PLUGINS_SEL. The comment above it ("tags that name THIS set and nothing else") is false once two runs share a pair. Phase C then writes the version tag with no check of what the image reports.
  • .github/scripts/resolve-platform.py resolve_images: an unarmed set's portal is resolved through pair_tag. The tester comes from its unique version tag. So the two halves of one set come from two runs.

Fix

  • Identity by run, not by pair: #6177 (open, another session) moves arm and the resolver to the per-run staging tag staging-<core7>-<run id> and retires the pair tag. That removes the race at its root.
  • Gates, so a contradicting set can never be armed or sealed again (PR to follow): (1) arm asserts on every architecture that the image it is about to arm reports the version it is armed as. (2) node-repo-publish-bake.yml refuses to publish when any bundle stamps a floor above the portal the publication names. (3) promote phase B never moves main/latest backwards.
  • Owed after both land (an operator decision, not done here): the armed tag memex-portal-ai:3.0.0-ci.9985 still names the 9984 image. Even with the staging-tag resolver, a consumer that reads the version tag first resolves set 9985 to the wrong portal. Re-pointing it at sha256:53aa780…, or letting a newer armed set supersede it, is what makes set 9985 consistent.

Not established

  • Whether memex's converged: false and its pods restarting through the evening are caused by the self-updater seeing a reported 9984 under an armed 9985 tag. I did not read the self-updater's log.
  • Whether any satellite other than Manufacturing consumed 9985 inconsistently. Reinsurance, SocialMedia and Crm were green on the same 12:27 dispatch. Education's red has a different cause (install gate, not the floor).
  • New sets have been stuck since ~14:00Z for an unrelated reason: every portal build fails on jobs.plugins-modules.with.modules is absent or empty (#6155). No consistent set newer than 9995 can appear until that is fixed.
Ngôn ngữ chính
C#
Star
12
Fork
5
Merge trung bình
5 giờ 10 phút
Pull request đã merge (30 ngày)
981

Chuẩn bị môi trường

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Systemorph/MeshWeaver

Tất cả issue của Systemorph/MeshWeaver

Issue tương tự

Thêm issue về C#

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.