Mesh writes fail pod-wide while a WhatsApp module service is unregistered — log-incident control plane blind for ~6 minutes

Start at TryLeaseModuleService in src/MeshWeaver.Mesh.Contract/ModuleContexts.cs:240, then ModuleServiceProxy.Create in ModuleServiceForwarders.cs:27, to see why mesh writes resolve the WhatsApp keyed service. The issue lists three competing causes and the logs needed to separate them, so the first step is reading the per-pod ModuleLoad logs and the current WhatsApp registrations. Done means one cause is confirmed and a fix is agreed with maintainers, which this issue does not yet support for a newcomer.

Observed

For roughly six minutes on the memex portal, every mesh write that routed through pod activation failed with Autofac.Core.Registration.ComponentNotRegisteredException for the keyed service mw-module-service:10 (MeshWeaver.WhatsApp.IWhatsAppRecentChats). Two user-facing symptoms were filed as child incidents at the same instant on the same pod:

• Admin/_LogIncident/99fe4b14ec1654b7 — an unhandled exception in a Blazor circuit while activating λ:MeshWeaver.Mesh.Services.IMeshService; the stack runs Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredKeyedService → MeshWeaver.Mesh.ModuleContexts.TryLeaseModuleService (src/MeshWeaver.Mesh.Contract/ModuleContexts.cs:240). So activation of IMeshService leases the WhatsApp module's keyed service — one missing module registration faults mesh-service activation generally, not only WhatsApp features.
• Admin/_LogIncident/b9b31f8eebcc322a — MeshWeaver.Graph.ActivityTracking failed to record activity for user rsalzmann on path TelefonicaSeguros/AiPartnerRfpEs (write to rsalzmann/_UserActivity/TelefonicaSeguros_AiPartnerRfpEs), "Inner UpdateNode faulted" with the same missing service — i.e. writes to namespaces unrelated to WhatsApp also go through the lease.

This ticket is the control-plane view of the same event: the log-incident control plane could not write to either child incident node during the window — no triage, no ticket, no recurrence comment — because its own deliveries failed with DeliveryFailureException carrying the same missing-service resolution failure. Writes to the incident nodes themselves therefore also require the module-service lease.

Where and when

One burst, one window, on the memex namespace; the control-plane report was emitted once per outage per incident node across all three portal pods. Correlations read during triage:

• Plugins/WhatsApp (Package, version 1056) and WhatsApp (Store/Plugin, version 119) were last modified 2026-10-16T00:12:51Z — about six minutes before the first failure.
• WhatsApp/Setup ("Setting up WhatsApp") was created 2026-10-15T14:45:58Z — WhatsApp was being set up the afternoon before.
• Prior art on this exact surface: an Ops/Logs entry from 2026-09-18 (memex-1789674127277972299) shows an unregistered type deserialised on hub WhatsApp; PRs Systemorph/MeshWeaver #6202 (multi-registration module services) and #6206 (module types served from the modules wherever asked), both 2026-10-07, changed how module services are keyed and served.

Impact

• One known user session crashed (rsalzmann's circuit died with an unhandled exception); activity tracking for that user's action on TelefonicaSeguros/AiPartnerRfpEs was not recorded.
• The observability pipeline lost its recording path for ~6 minutes: decisions the control plane made in that window were not persisted to the incident nodes (the mechanism re-makes attempts, so the window is a delay/visibility gap rather than proven data loss).
• The blast radius is wider than WhatsApp: a single module's missing keyed service blocked IMeshService activation and node writes generally on the affected pod.

Not established

• Whether the WhatsApp module was mid-reload when the failures began (a transient live-update window) or whether it loaded without registering IWhatsAppRecentChats durably. The control plane reports once per outage, so the absence of later reports does not prove recovery; as of triage time the service's current registration state was not read.
• Why a write to Admin/_LogIncident/* or rsalzmann/_UserActivity/* requires the WhatsApp module's service at all — whether message delivery/serialisation unconditionally leases every loaded module's keyed services, or whether something in the WhatsApp setup specifically wired IWhatsAppRecentChats into those paths. Not determined from the available stacks.
• No [ModuleLoad] MeshWeaver.WhatsApp line was found in Ops/Logs for the 00:12–00:26 window (the newest found is 2026-10-07), so whether a reload ran in that window is unknown.
• Whether pods -8fg4t and -wxxdc had circuit crashes too, or only the delivery failures; the two child bursts are both from -dmbcw.
• Whether this is a recurrence of the same defect the two child incidents represent — same missing service, same instant, same pod — or a distinct failure of the delivery path. The signatures coincide exactly.

Candidate causes and the reading that separates them

1. Transient live-update window during the WhatsApp package landing at 00:12:51 — the module's keyed service was absent while the module was being swapped, and became registered again ~6 minutes later. Predicts a [ModuleLoad] MeshWeaver.WhatsApp line (or self-update intake entry) inside the window on each pod and that the same failure appears on other modules during their future reloads. Reading: the per-pod [ModuleLoad]/[SelfUpdate] logs for 2026-10-16 00:12–00:26.
2. The module loaded but never registered IWhatsAppRecentChats (first real activation after the previous afternoon's WhatsApp setup; new plugin surface). Predicts the exception recurs on every resolve attempt and that the service is still absent now. Reading: attempt one mesh write / circuit activation on memex now and check the current WhatsApp registrations in Ops/Modules/memex (module inventory).
3. Keyed-index drift — mw-module-service:10 is a positional key; a change in the module inventory could leave index 10 naming a registration that no longer exists while WhatsApp's actual registration moved. Predicts other modules failing at other indexes and correlation with Ops/Modules/memex inventory changes (last updated 2026-10-07, version 1895). Reading: compare the failing key's index against the current module inventory ordering.

The 2026-09-18 "Unregistered type … on hub WhatsApp" entry makes cause 2/3-flavoured registration mismatch plausible, but none of the three is established by this evidence.

Where to look

• MeshWeaver.Mesh.ModuleContexts.TryLeaseModuleService — src/MeshWeaver.Mesh.Contract/ModuleContexts.cs:240 (from the child incident's stack).
• MeshWeaver.Mesh.ModuleServiceProxy.Create — src/MeshWeaver.Mesh.Contract/ModuleServiceForwarders.cs:27 (from the 2026-10-06 stack line Ops/Logs/memex-1791314750821267916-fd4594-qgxdl).
• Child incidents Admin/_LogIncident/99fe4b14ec1654b7 and Admin/_LogIncident/b9b31f8eebcc322a — same underlying signature; whichever of the three tickets carries the fix, the other two can be closed against it.
• Ops/Modules/memex (module inventory), Plugins/WhatsApp / WhatsApp (Store/Plugin) package timeline.

Severity is judged from the numbers: one recovered six-minute burst, one known user-visible circuit crash, and a six-minute observability recording gap — sev:M. If cause 1 or 3 holds, the same blast radius (all mesh writes on a pod) recurs on every module reload, which a human may re-rule upward once the discriminating reading is taken.

Evidence

2026-10-16 00:25:50Z memex-portal-deployment-65b9d48c85-dmbcw fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident b9b31f8eebcc322a: Admin/_LogIncident/b9b31f8eebcc322a has not been writable for 00:05:59.3353751, so nothing this control plane decided in that time is recorded on it — no triage, no ticket, no recurrence comment. Every attempt is re-made when its trigger fires; this is reported once per outage. Latest failure: DeliveryFailureException: Path resolution for 'Admin/_LogIncident/b9b31f8eebcc322a' failed: The requested service 'mw-module-service:10 (MeshWeaver.WhatsApp.IWhatsAppRecentChats)' has not been registered. To avoid this exception, either register a component to provide the service, check for service registration using IsRegistered(), or use the ResolveOptional() method to resolve an optional dependency.
      
      See https://autofac.rtfd.io/help/service-not-registered for more info. — the stack trace is on the per-attempt warnings.
2026-10-16 00:25:50Z memex-portal-deployment-65b9d48c85-8fg4t fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident b9b31f8eebcc322a: Admin/_LogIncident/b9b31f8eebcc322a has not been writable for 00:05:59.2899414, so nothing this control plane decided in that time is recorded on it — no triage, no ticket, no recurrence comment. Every attempt is re-made when its trigger fires; this is reported once per outage. Latest failure: DeliveryFailureException: Path resolution for 'Admin/_LogIncident/b9b31f8eebcc322a' failed: The requested service 'mw-module-service:10 (MeshWeaver.WhatsApp.IWhatsAppRecentChats)' has not been registered. To avoid this exception, either register a component to provide the service, check for service registration using IsRegistered(), or use the ResolveOptional() method to resolve an optional dependency.
      
      See https://autofac.rtfd.io/help/service-not-registered for more info. — the stack trace is on the per-attempt warnings.
2026-10-16 00:25:50Z memex-portal-deployment-65b9d48c85-8fg4t fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 99fe4b14ec1654b7: Admin/_LogIncident/99fe4b14ec1654b7 has not been writable for 00:06:00.0162121, so nothing this control plane decided in that time is recorded on it — no triage, no ticket, no recurrence comment. Every attempt is re-made when its trigger fires; this is reported once per outage. Latest failure: DeliveryFailureException: Path resolution for 'Admin/_LogIncident/99fe4b14ec1654b7' failed: The requested service 'mw-module-service:10 (MeshWeaver.WhatsApp.IWhatsAppRecentChats)' has not been registered. To avoid this exception, either register a component to provide the service, check for service registration using IsRegistered(), or use the ResolveOptional() method to resolve an optional dependency.
      
      See https://autofac.rtfd.io/help/service-not-registered for more info. — the stack trace is on the per-attempt warnings.
2026-10-16 00:25:50Z memex-portal-deployment-65b9d48c85-wxxdc fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident b9b31f8eebcc322a: Admin/_LogIncident/b9b31f8eebcc322a has not been writable for 00:05:59.1762103, so nothing this control plane decided in that time is recorded on it — no triage, no ticket, no recurrence comment. Every attempt is re-made when its trigger fires; this is reported once per outage. Latest failure: DeliveryFailureException: Path resolution for 'Admin/_LogIncident/b9b31f8eebcc322a' failed: The requested service 'mw-module-service:10 (MeshWeaver.WhatsApp.IWhatsAppRecentChats)' has not been registered. To avoid this exception, either register a component to provide the service, check for service registration using IsRegistered(), or use the ResolveOptional() method to resolve an optional dependency.
      
      See https://autofac.rtfd.io/help/service-not-registered for more info. — the stack trace is on the per-attempt warnings.
2026-10-16 00:25:50Z memex-portal-deployment-65b9d48c85-wxxdc fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 99fe4b14ec1654b7: Admin/_LogIncident/99fe4b14ec1654b7 has not been writable for 00:06:00.0036747, so nothing this control plane decided in that time is recorded on it — no triage, no ticket, no recurrence comment. Every attempt is re-made when its trigger fires; this is reported once per outage. Latest failure: DeliveryFailureException: Path resolution for 'Admin/_LogIncident/99fe4b14ec1654b7' failed: The requested service 'mw-module-service:10 (MeshWeaver.WhatsApp.IWhatsAppRecentChats)' has not been registered. To avoid this exception, either register a component to provide the service, check for service registration using IsRegistered(), or use the ResolveOptional() method to resolve an optional dependency.
      
      See https://autofac.rtfd.io/help/service-not-registered for more info. — the stack trace is on the per-attempt warnings.

_{Opened automatically from Admin/_LogIncident/0d37dd083aac26d0. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site c6c21af6b662b49b: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}