Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Registry instance-key resolution outage (503) defers boot reconcile of installed plugins — PENDING until next successful feed read

Closed
  • Is something wrong?
#6,450 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp, kubernetes
Domain
devops, observability-sre

Research direction

Start with src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs at the ListCatalog failure path and inspect MeshWeaver.PluginCatalog.RegistryUpdateReconciler plus the pending entry on Plugins/_RegistryReconcileLedger. Correlate the incident timestamp with registry and instance-key-resolution service logs, and verify the ledger clears after the next successful catalog read or install. Done means the registry outage is identified or ruled out, the pending reconcile is confirmed cleared, and any recurring fault has an agreed owner and fix.

Written by the indexing model from the issue text.

Description

bug sev:L

What is failing

The boot-time RegistryUpdateReconciler could not read the package feed of the Plugins registry (https://memex.meshweaver.cloud), so installed packages from that registry were not reconciled this boot. The registry answered the catalog-list call with HTTP 503 — Instance-key resolution is temporarily unavailable — on every attempt through the full retry window, and the reconciler recorded the reconcile as PENDING on Plugins/_RegistryReconcileLedger instead.

Probable cause (medium confidence)

A transient availability problem in the registry-side instance-key resolution service — i.e. on the server hosting memex.meshweaver.cloud, not in the client. The client side behaved as designed: it retried, then degraded gracefully to a durable PENDING ledger entry that the next successful feed read (a catalog open or an install) will pick up. The server's own error text (retry shortly… NOT a statement about your key or your grant) indicates this is an anticipated transient condition on the registry.

What the evidence does not show: whether this was a brief registry blip, a cold-start race on the freshly started portal pod, or the start of a persistent instance-key-resolution outage. There is exactly one occurrence on one pod, and the reconciler does not re-log while a ledger entry is pending, so absence of further lines proves nothing either way.

Impact

Minimal as evidenced: a single occurrence on a single pod, self-healing by design — the reconcile is deferred, not lost, and runs on the next successful feed read. No user-visible failure is in evidence. The risk to watch is that PENDING entries only clear on a successful feed read: if instance-key resolution on the registry stays down, every catalog open and install against this registry fails too, which would escalate this from nuisance to outage.

Where to look

  • Client (behaved correctly, for context): MeshWeaver.PluginCatalog.RegistryPackageSource.ListCatalog — src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs (throw site per stack trace), and MeshWeaver.PluginCatalog.RegistryUpdateReconciler.
  • Durable state: the ledger entry on Plugins/_RegistryReconcileLedger for https://memex.meshweaver.cloud (pending: true, attempts: 4, fault text matching this incident). Verify it clears on the next successful feed read — if it is still pending hours later, the registry-side outage is persistent and this ticket should be re-ruled upward.
  • Root cause lives on the registry service serving https://memex.meshweaver.cloud: check the health/availability of its instance-key resolution component around the incident window. Also note this is not the first RegistryUpdateReconciler/RegistryPackageSource warning in the ops logs this week — worth checking whether those earlier warnings share this fault text before treating this as isolated.

Severity

Filed as sev:L: rare, single occurrence, graceful degradation with a durable, self-healing fallback. If the ledger entry does not clear or the fault repeats, re-rule to sev:M/sev:H — a registry whose instance-key resolution is persistently down blocks installs and catalog opens for every tenant of that registry.

Evidence
Fingerprint 70c30a5bc577c89b
Category MeshWeaver.PluginCatalog.RegistryUpdateReconciler
Severity Error
Exception MeshWeaver.PluginCatalog.RegistryResponseException
Top frame MeshWeaver.PluginCatalog.RegistryPackageSource.<>c__DisplayClass16_0.<<ListCatalog>b__0>d.MoveNext()
Namespace memex
Pods memex-portal-deployment-64d884c879-gqsx9
Occurrences 1
First seen 2026-10-10 18:16:06Z
Last seen 2026-10-10 18:16:06Z
Routing not determined — no configured route matches the category MeshWeaver.PluginCatalog.RegistryUpdateReconciler. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-10 18:16:06Z memex-portal-deployment-64d884c879-gqsx9 fail: MeshWeaver.PluginCatalog.RegistryUpdateReconciler[0]
      [RegistryUpdate] could not read the package feed of Plugins (https://memex.meshweaver.cloud) after 4 attempt(s) — installed packages from it are NOT reconciled this boot. Recorded as PENDING on Plugins/_RegistryReconcileLedger; the next successful feed read against this registry (a catalog open, an install) runs the reconcile. Platform admins have been notified.
      MeshWeaver.PluginCatalog.RegistryResponseException: Registry catalog list failed (503): {"error":"Instance-key resolution is temporarily unavailable — retry shortly. This is NOT a statement about your key or your grant."}
         at MeshWeaver.PluginCatalog.RegistryPackageSource.<>c__DisplayClass16_0.<<ListCatalog>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.PluginCatalog/RegistryPackageSource.cs:line 103
      --- End of stack trace from previous location ---
         at MeshWeaver.Mesh.Threading.IoPool.<>c__DisplayClass55_1`1.<<InvokeCore>b__2>d.MoveNext()

Opened automatically from Admin/_LogIncident/70c30a5bc577c89b. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 1fe670af0c57dcb4: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
4h 19m
Merged PRs (30d)
932

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.