Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Registry instance-key resolution outage (503) defers boot reconcile of installed plugins — PENDING until next successful feed read

Chiusa
  • Qualcosa non va?
#6,450 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp, kubernetes
Ambito
devops, observability-sre

Direzione di ricerca

Start with src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs at the ListCatalog failure path and inspect MeshWeaver.PluginCatalog.RegistryUpdateReconciler plus the pending entry on Plugins/_RegistryReconcileLedger. Correlate the incident timestamp with registry and instance-key-resolution service logs, and verify the ledger clears after the next successful catalog read or install. Done means the registry outage is identified or ruled out, the pending reconcile is confirmed cleared, and any recurring fault has an agreed owner and fix.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:L

What is failing

The boot-time RegistryUpdateReconciler could not read the package feed of the Plugins registry (https://memex.meshweaver.cloud), so installed packages from that registry were not reconciled this boot. The registry answered the catalog-list call with HTTP 503 — Instance-key resolution is temporarily unavailable — on every attempt through the full retry window, and the reconciler recorded the reconcile as PENDING on Plugins/_RegistryReconcileLedger instead.

Probable cause (medium confidence)

A transient availability problem in the registry-side instance-key resolution service — i.e. on the server hosting memex.meshweaver.cloud, not in the client. The client side behaved as designed: it retried, then degraded gracefully to a durable PENDING ledger entry that the next successful feed read (a catalog open or an install) will pick up. The server's own error text (retry shortly… NOT a statement about your key or your grant) indicates this is an anticipated transient condition on the registry.

What the evidence does not show: whether this was a brief registry blip, a cold-start race on the freshly started portal pod, or the start of a persistent instance-key-resolution outage. There is exactly one occurrence on one pod, and the reconciler does not re-log while a ledger entry is pending, so absence of further lines proves nothing either way.

Impact

Minimal as evidenced: a single occurrence on a single pod, self-healing by design — the reconcile is deferred, not lost, and runs on the next successful feed read. No user-visible failure is in evidence. The risk to watch is that PENDING entries only clear on a successful feed read: if instance-key resolution on the registry stays down, every catalog open and install against this registry fails too, which would escalate this from nuisance to outage.

Where to look

  • Client (behaved correctly, for context): MeshWeaver.PluginCatalog.RegistryPackageSource.ListCatalog — src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs (throw site per stack trace), and MeshWeaver.PluginCatalog.RegistryUpdateReconciler.
  • Durable state: the ledger entry on Plugins/_RegistryReconcileLedger for https://memex.meshweaver.cloud (pending: true, attempts: 4, fault text matching this incident). Verify it clears on the next successful feed read — if it is still pending hours later, the registry-side outage is persistent and this ticket should be re-ruled upward.
  • Root cause lives on the registry service serving https://memex.meshweaver.cloud: check the health/availability of its instance-key resolution component around the incident window. Also note this is not the first RegistryUpdateReconciler/RegistryPackageSource warning in the ops logs this week — worth checking whether those earlier warnings share this fault text before treating this as isolated.

Severity

Filed as sev:L: rare, single occurrence, graceful degradation with a durable, self-healing fallback. If the ledger entry does not clear or the fault repeats, re-rule to sev:M/sev:H — a registry whose instance-key resolution is persistently down blocks installs and catalog opens for every tenant of that registry.

Evidence
Fingerprint 70c30a5bc577c89b
Category MeshWeaver.PluginCatalog.RegistryUpdateReconciler
Severity Error
Exception MeshWeaver.PluginCatalog.RegistryResponseException
Top frame MeshWeaver.PluginCatalog.RegistryPackageSource.<>c__DisplayClass16_0.<<ListCatalog>b__0>d.MoveNext()
Namespace memex
Pods memex-portal-deployment-64d884c879-gqsx9
Occurrences 1
First seen 2026-10-10 18:16:06Z
Last seen 2026-10-10 18:16:06Z
Routing not determined — no configured route matches the category MeshWeaver.PluginCatalog.RegistryUpdateReconciler. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-10 18:16:06Z memex-portal-deployment-64d884c879-gqsx9 fail: MeshWeaver.PluginCatalog.RegistryUpdateReconciler[0]
      [RegistryUpdate] could not read the package feed of Plugins (https://memex.meshweaver.cloud) after 4 attempt(s) — installed packages from it are NOT reconciled this boot. Recorded as PENDING on Plugins/_RegistryReconcileLedger; the next successful feed read against this registry (a catalog open, an install) runs the reconcile. Platform admins have been notified.
      MeshWeaver.PluginCatalog.RegistryResponseException: Registry catalog list failed (503): {"error":"Instance-key resolution is temporarily unavailable — retry shortly. This is NOT a statement about your key or your grant."}
         at MeshWeaver.PluginCatalog.RegistryPackageSource.<>c__DisplayClass16_0.<<ListCatalog>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs:line 103
      --- End of stack trace from previous location ---
         at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass55_1`1.<<InvokeCore>b__2>d.MoveNext()

Opened automatically from Admin/_LogIncident/70c30a5bc577c89b. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 1fe670af0c57dcb4: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
4h 50m
PR unite (30g)
949

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.