Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Operator lane not live on memex portal — every fleet provisioning plan aborts at 'Launch operator job'

Open
  • Is something wrong?
#6,896 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp, helm, kubernetes
Domain
devops, infrastructure

Research direction

Start with Hosting/Skill/instance and its refusal table and live-probe procedure; then inspect the memex fleet config values and Helm release history for the memex namespace across the reported window. Check Ops/Actions/logs-memex-20260923-operator-disabled and whether an existing issue tracks this recurring state. Done means explaining why the rolled portal replicas still report operator actions disabled and identifying the required enablement fix.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

Every provisioning plan run on the memex portal — for the fleet's own instances control, memex-cloud and memex — aborts at its Launch operator job phase: the plan fail-closes because instance actions are disabled on this installation (Hosting:Operator:Enabled is not in effect on the executing portal). The whole sanctioned instance-action lane (Provision, Backup, Suspend, Roll, Restart) is refused on this installation, on every replica, and the failures are still being produced now.

Probable cause (medium-high confidence)

The operator enablement is not live on the memex deployment. Two readings, and the incident alone cannot distinguish them:

  1. Enablement missing/not applied — the values block for this deployment does not carry Hosting:Operator:Enabled=true (plus the operator image, namespace and token), or the enabling Helm release run never happened. This is the documented trap: the /instance skill warns explicitly "do not trust the repos to answer 'is the operator live?'" — values reach a container only when the Helm release rolls, and an enablement can be fully merged and still refuse (precedent 2026-08-22). Supporting evidence: this exact refusal has been seen on this installation before — a self-update roll was stamped Unobserved with this sentence on 2026-09-23 (Ops/Actions/logs-memex-20260923-operator-disabled, MeshWeaver#5544 / Plugins#2178 context) — so the enablement state on memex is fragile, not a one-off.
  2. Enablement present but not effective — the values are correct in the release but not read/bound by the portal. Arguing against a stale-replica race: the portal's replica set changed mid-window (pod template rolled between the 2026-10-07 evening and 2026-10-08 morning samples) and the new replicas still refuse, so whatever rolled did not make the operator live.

Either way, the defect-shaped question for the engineer is: why is the operator lane not live on a portal whose release just rolled, and why does the same installation keep lapsing into this state?

Impact

Fleet lifecycle automation is down on this installation: provisioning of control, memex-cloud and memex cannot complete, and any action-dependent flow (self-update rolls, self-service instance requests, backups) will refuse or be stamped Unobserved. The failure is fail-closed by design — no corruption, plans are idempotent and re-runnable — and the only alternative today is the documented break-glass kubectl scale, which the runbook itself marks as creating reconciliation debt (no dump, no paywall, no audit record).

Where to look

  • The refusal semantics and the live-probe procedure: Hosting/Skill/instance — the "Do not trust the repos…" section and the refusal table; the probe is a real Backup action read at its Launch operator job phase (never a dry run — the dry-run plan ends before the operator phase and proves nothing).
  • The executed enablement runbook Hosting/OperatorEnablement.md (referenced by the skill; not present in this mesh instance — in the fleet config repo).
  • The memex fleet config repo: values for Hosting:Operator:Enabled and the Helm release history for the memex namespace across this window — did the release that rolled mid-window carry the operator values?
  • Prior recurrence on this installation: Ops/Actions/logs-memex-20260923-operator-disabled (2026-09-23, MeshWeaver#5544 / Plugins#2178 context) — worth checking whether an enablement issue already tracks the memex operator state before doing new work.

Not a duplicate of Systemorph/MeshWeaver#6117 (same ProvisionPlan category, but a Postgres query-provider stall at phase 2 — an unrelated defect).

Severity note: filed sev:M — a clear defect in the installation's enablement state with an easy workaround (run the enabling Helm release / restore the values). If it turns out the values are correct in the rolled release and still not effective (a code/config-binding defect), or if customer-facing instance provisioning is being blocked at scale, a human should re-rule upward.

Evidence
Fingerprint a5831011a7fa1c78
Category ProvisionPlan
Severity Error
Namespace memex
Pods memex-portal-deployment-dcf6f66d5-5khwq, memex-portal-deployment-dcf6f66d5-k2ntx, memex-portal-deployment-74b8fd86c9-qf676, memex-portal-deployment-8cc8dc5ff-6bjzm, memex-portal-deployment-6844bc6c5-grbh9, memex-portal-deployment-5f974b4649-w5jnb, memex-portal-deployment-69fdc8c44-hv4rk, memex-portal-deployment-cfb58787b-b6klj, memex-portal-deployment-8fc6bfb94-x8zh9, memex-portal-deployment-6b67596696-p57qd, memex-portal-deployment-5b885f54df-b9nj9, memex-portal-deployment-59f889d6dd-d9n6x, memex-portal-deployment-588fc6b6b7-ljjsl, memex-portal-deployment-769f777788-99vt2, memex-portal-deployment-7d79f7db8f-8rmj9, memex-portal-deployment-7b55d9cd97-62jkm, memex-portal-deployment-d49477bf6-lkvl6, memex-portal-deployment-795dcbbcd6-mgkdq, memex-portal-deployment-795dcbbcd6-rmcz2, memex-portal-deployment-6bd66fccfd-gh82p
Occurrences 39
First seen 2026-10-07 16:02:51Z
Last seen 2026-10-17 06:53:00Z
Routing not determined — no configured route matches the category ProvisionPlan. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-09 19:46:17Z memex-portal-deployment-59f889d6dd-d9n6x fail: ProvisionPlan[0]
      [ProvisionPlan] 'build' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 19:47:03Z memex-portal-deployment-588fc6b6b7-ljjsl fail: ProvisionPlan[0]
      [ProvisionPlan] 'control' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 19:47:03Z memex-portal-deployment-588fc6b6b7-ljjsl fail: ProvisionPlan[0]
      [ProvisionPlan] 'build' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 21:48:54Z memex-portal-deployment-769f777788-99vt2 fail: ProvisionPlan[0]
      [ProvisionPlan] 'control' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-09 21:51:45Z memex-portal-deployment-7d79f7db8f-8rmj9 fail: ProvisionPlan[0]
      [ProvisionPlan] 'control' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-10 00:42:55Z memex-portal-deployment-7b55d9cd97-62jkm fail: ProvisionPlan[0]
      [ProvisionPlan] 'pearl' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-10 00:43:18Z memex-portal-deployment-d49477bf6-lkvl6 fail: ProvisionPlan[0]
      [ProvisionPlan] 'pearl' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-12 04:21:42Z memex-portal-deployment-795dcbbcd6-mgkdq fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex-cloud' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-12 04:21:43Z memex-portal-deployment-795dcbbcd6-rmcz2 fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 4/8 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)
2026-10-17 06:53:00Z memex-portal-deployment-6bd66fccfd-gh82p fail: ProvisionPlan[0]
      [ProvisionPlan] 'pg3' FAILED in phase 4/6 'Launch operator job': instance actions are disabled on this installation — set Hosting:Operator:Enabled=true on the control instance (and ONLY there: the operator identity can delete namespaces and drop databases)

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/5717, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/a5831011a7fa1c78. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8aa2f9ea523cf69c: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 51m
Merged PRs (30d)
970

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.