Fleet coordinator cannot create its Hosting/Coordinator node — the runtime fault-loops every minute and no coordinator round has ever run

Cerrado
  • ¿Algo no está bien?
#6,151 5 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
csharp
Área
backend

Línea de trabajo

Read Hosting/Coordination/Source/FleetCoordinatorRuntime.cs, focusing on EnsureNode and AddFleetCoordinator's SelfHealing wrapper. Compare the deployment and closed-type registration details in the referenced #2871 and #2898 pull-request nodes. Verify whether the node creation refusal is surfaced distinctly and whether a missing node still causes repeated schedule restarts; done means the deployment gap or runtime failure is clearly reported and the one-minute fault loop is addressed.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug sev:M

What is failing

The fleet coordinator (Plugins #2871, merged 2026-10-05 08:52Z) is armed on the control instance's always-on Hosting/PlatformBuilds hub, but it cannot bring up its own node Hosting/Coordinator. Every minute the schedule faults with MeshWeaver.Messaging.DeliveryFailureException — [ADDRESS] node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting'" — and restarts, so the coordinator has never completed a single round: no startup round, no fleet ranking, no dispatch, since it was deployed.

Probable cause (high confidence)

Hosting/Coordinator is created at run time with NodeType Hosting/Coordination, and that type is not usable on the running control instance:

  • The NodeType definition node does not exist in the mesh (no Hosting/Coordination anywhere under Hosting — every sibling type like Hosting/Queue, Hosting/RepoHealth is present), so [PERSON_NAME]Hostingthat was to carryHosting/Coordination.json` (#2871's deploy step 1) appears not to have landed.
  • Even with a sync, the control instance runs Mesh:ClosedTypeSet, and #2871's registration of Hosting/Coordination in src/MeshWeaver.Fleet.Control/Generated/FleetControlNodeTypes.g.cs only takes effect after the control image is rebuilt and Rolled — exactly the state PR #2898 documented for Hosting/PrWatch: "until then the running control pods keep the old registration set", and an unregistered type is refused at activation.

The aggravating code defect: FleetCoordinatorRuntime.EnsureNode wraps the create in .Catch(_ => Observable.Return(Unit.Default)) and swallows the refusal silently. The round loop then combines over GetMeshNodeStream("Hosting/Coordinator"), which faults, and the SelfHealing(..., TimeSpan.FromMinutes(1)) wrapper logs the generic "the schedule faulted… restarting in a minute" line — the one-per-minute cadence in this incident is that self-healing restart. Nothing in the logs distinguishes "the create was refused" from any other fault, so the runtime spins forever instead of surfacing the deployment gap.

Impact

The coordinator is a brand-new automation merged this morning; nothing that worked before is broken by its absence. What is affected: the coordinator's whole function (priority ranking and load-aware dispatch across the fleet's triage work) is dead on the control instance, and the control pod emits one error per minute indefinitely. Hosting:Coordinator:Enabled=false stops the noise if needed. Judged from the numbers this is a clear defect with an easy workaround, not an outage.

Where to look

  • Hosting/Triage/pull-request/systemorph-meshweaver-plugins-2871 — the coordinator's design and deploy steps: [PERSON_NAME] Hosting, recycle Hosting/PlatformBuildInbox and Hosting/Coordination. Note Hosting/Coordination is absent from the mesh while the runtime is clearly armed — the deploy is half-done.
  • Hosting/Triage/pull-request/systemorph-meshweaver-plugins-2898 — the same class of failure (an unregistered NodeType refused under the closed set) for Hosting/PrWatch, fixed by registering the type + image rebuild + Roll.
  • Hosting/Coordination/Source/FleetCoordinatorRuntime.cs — EnsureNode (the silent .Catch on the create) and AddFleetCoordinator's SelfHealing wrapper that turns the fault into the per-minute restart.

Suggested fix: complete the deploy (sync/register Hosting/Coordination, rebuild and roll the control image per #2898's pattern), and independently make EnsureNode log the create failure distinctly and back off instead of letting the loop fault-loop on a missing node — so a half-deployed coordinator says so instead of impersonating a schedule bug.

Evidence
Fingerprint 14487921ad43ddc7
Category FleetCoordinator
Severity Error
Exception MeshWeaver.Messaging.DeliveryFailureException
Namespace memex
Pods memex-portal-deployment-7c4764b686-tbr7f, memex-portal-deployment-58475df96-nprkt
Occurrences 17
First seen 2026-10-05 12:05:43Z
Last seen 2026-10-05 12:20:11Z
Routing not determined — no configured route matches the category FleetCoordinator. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 12:12:44Z memex-portal-deployment-7c4764b686-tbr7f fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:13:44Z memex-portal-deployment-7c4764b686-tbr7f fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:14:44Z memex-portal-deployment-7c4764b686-tbr7f fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:14:55Z memex-portal-deployment-58475df96-nprkt fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on content-type-registration/e147a6e18ee84db4a200198c414b4547 — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:15:11Z memex-portal-deployment-58475df96-nprkt fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:16:11Z memex-portal-deployment-58475df96-nprkt fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:17:11Z memex-portal-deployment-58475df96-nprkt fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:18:11Z memex-portal-deployment-58475df96-nprkt fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:19:11Z memex-portal-deployment-58475df96-nprkt fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').
2026-10-05 12:20:11Z memex-portal-deployment-58475df96-nprkt fail: FleetCoordinator[0]
      [Coordinator] the schedule faulted on Hosting/PlatformBuilds — restarting in a minute
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Hosting/Coordinator'. Closest ancestor is 'Hosting' (remainder='Coordinator').

Opened automatically from Admin/_LogIncident/14487921ad43ddc7. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site facaac0ee8f27d84: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lenguaje dominante
C#
Estrellas
12
Forks
5
Merge medio
5 h 10 min
PR fusionados (30 d)
979

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Systemorph/MeshWeaver

Todos los issues de Systemorph/MeshWeaver

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.