Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Operator lane not live on memex portal — every fleet provisioning plan aborts at 'Launch operator job'

Abierto
  • ¿Algo no está bien?
#6,896 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
csharp, helm, kubernetes
Área
devops, infrastructure

Línea de trabajo

Start with Hosting/Skill/instance and its refusal table and live-probe procedure; then inspect the memex fleet config values and Helm release history for the memex namespace across the reported window. Check Ops/Actions/logs-memex-20260923-operator-disabled and whether an existing issue tracks this recurring state. Done means explaining why the rolled portal replicas still report operator actions disabled and identifying the required enablement fix.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug sev:M

What is failing

Every provisioning plan run on the memex portal — for the fleet's own instances control, memex-cloud and memex — aborts at its Launch operator job phase: the plan fail-closes because instance actions are disabled on this installation (Hosting:Operator:Enabled is not in effect on the executing portal). The whole sanctioned instance-action lane (Provision, Backup, Suspend, Roll, Restart) is refused on this installation, on every replica, and the failures are still being produced now.

Probable cause (medium-high confidence)

The operator enablement is not live on the memex deployment. Two readings, and the incident alone cannot distinguish them:

  1. Enablement missing/not applied — the values block for this deployment does not carry Hosting:Operator:Enabled=true (plus the operator image, namespace and token), or the enabling Helm release run never happened. This is the documented trap: the /instance skill warns explicitly "do not trust the repos to answer 'is the operator live?'" — values reach a container only when the Helm release rolls, and an enablement can be fully merged and still refuse (precedent 2026-08-22). Supporting evidence: this exact refusal has been seen on this installation before — a self-update roll was stamped Unobserved with this sentence on 2026-09-23 (Ops/Actions/logs-memex-20260923-operator-disabled, MeshWeaver#5544 / Plugins#2178 context) — so the enablement state on memex is fragile, not a one-off.
  2. Enablement present but not effective — the values are correct in the release but not read/bound by the portal. Arguing against a stale-replica race: the portal's replica set changed mid-window (pod template rolled between the 2026-10-07 evening and 2026-10-08 morning samples) and the new replicas still refuse, so whatever rolled did not make the operator live.

Either way, the defect-shaped question for the engineer is: why is the operator lane not live on a portal whose release just rolled, and why does the same installation keep lapsing into this state?

Impact

Fleet lifecycle automation is down on this installation: provisioning of control, memex-cloud and memex cannot complete, and any action-dependent flow (self-update rolls, self-service instance requests, backups) will refuse or be stamped Unobserved. The failure is fail-closed by design — no corruption, plans are idempotent and re-runnable — and the only alternative today is the documented break-glass kubectl scale, which the runbook itself marks as creating reconciliation debt (no dump, no paywall, no audit record).

Where to look

  • The refusal semantics and the live-probe procedure: Hosting/Skill/instance — the "Do not trust the repos…" section and the refusal table; the probe is a real Backup action read at its Launch operator job phase (never a dry run — the dry-run plan ends before the operator phase and proves nothing).
  • The executed enablement runbook Hosting/OperatorEnablement.md (referenced by the skill; not present in this mesh instance — in the fleet config repo).
  • The memex fleet config repo: values for Hosting:Operator:Enabled and the Helm release history for the memex namespace across this window — did the release that rolled mid-window carry the operator values?
  • Prior recurrence on this installation: Ops/Actions/logs-memex-20260923-operator-disabled (2026-09-23, MeshWeaver#5544 / Plugins#2178 context) — worth checking whether an enablement issue already tracks the memex operator state before doing new work.

Not a duplicate of Systemorph/MeshWeaver#6117 (same ProvisionPlan category, but a Postgres query-provider stall at phase 2 — an unrelated defect).

Severity note: filed sev:M — a clear defect in the installation's enablement state with an easy workaround (run the enabling Helm release / restore the values). If it turns out the values are correct in the rolled release and still not effective (a code/config-binding defect), or if customer-facing instance provisioning is being blocked at scale, a human should re-rule upward.

Evidence
Fingerprint a5831011a7fa1c78
Category ProvisionPlan
Severity Error
Namespace memex
Pods memex-portal-deployment-dcf6f66d5-5khwq, memex-portal-deployment-dcf6f66d5-k2ntx, memex-portal-deployment-74b8fd86c9-qf676, memex-portal-deployment-8cc8dc5ff-6bjzm, memex-portal-deployment-6844bc6c5-grbh9, memex-portal-deployment-5f974b4649-w5jnb, memex-portal-deployment-69fdc8c44-hv4rk, memex-portal-deployment-cfb58787b-b6klj, memex-portal-deployment-8fc6bfb94-x8zh9, memex-portal-deployment-6b67596696-p57qd, memex-portal-deployment-5b885f54df-b9nj9, memex-portal-deployment-59f889d6dd-d9n6x, memex-portal-deployment-588fc6b6b7-ljjsl, memex-portal-deployment-769f777788-99vt2, memex-portal-deployment-7d79f7db8f-8rmj9, memex-portal-deployment-7b55d9cd97-62jkm, memex-portal-deployment-d49477bf6-lkvl6, memex-portal-deployment-795dcbbcd6-mgkdq, memex-portal-deployment-795dcbbcd6-rmcz2, memex-portal-deployment-6bd66fccfd-gh82p
Occurrences 39
First seen 2026-10-07 16:02:51Z
Last seen 2026-10-17 06:53:00Z
Routing not determined — no configured route matches the category ProvisionPlan. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-09 19:46:17Z memex-portal-deployment-59f889d6dd-d9n6x fail: ProvisionPlan[0]
      [ProvisionPlan] 'build' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 19:47:03Z memex-portal-deployment-588fc6b6b7-ljjsl fail: ProvisionPlan[0]
      [ProvisionPlan] 'control' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 19:47:03Z memex-portal-deployment-588fc6b6b7-ljjsl fail: ProvisionPlan[0]
      [ProvisionPlan] 'build' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 21:48:54Z memex-portal-deployment-769f777788-99vt2 fail: ProvisionPlan[0]
      [ProvisionPlan] 'control' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 21:51:45Z memex-portal-deployment-7d79f7db8f-8rmj9 fail: ProvisionPlan[0]
      [ProvisionPlan] 'control' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-10 00:42:55Z memex-portal-deployment-7b55d9cd97-62jkm fail: ProvisionPlan[0]
      [ProvisionPlan] 'pearl' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-10 00:43:18Z memex-portal-deployment-d49477bf6-lkvl6 fail: ProvisionPlan[0]
      [ProvisionPlan] 'pearl' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-12 04:21:42Z memex-portal-deployment-795dcbbcd6-mgkdq fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex-cloud' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-12 04:21:43Z memex-portal-deployment-795dcbbcd6-rmcz2 fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-17 06:53:00Z memex-portal-deployment-6bd66fccfd-gh82p fail: ProvisionPlan[0]
      [ProvisionPlan] 'pg3' FAILED in phase 4/6 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/5717, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/a5831011a7fa1c78. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8aa2f9ea523cf69c: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lenguaje dominante
C#
Estrellas
12
Forks
5
Merge medio
5 h 51 min
PR fusionados (30 d)
970

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Systemorph/MeshWeaver

Todos los issues de Systemorph/MeshWeaver

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.