Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

ProvisionPlan log-ingestion run drops user identity on write-back (0 of 47 entries landed) and its Loki query phase also times out / hits 429

Closed
  • Is something wrong?
#5,717 127 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp, kubernetes
Domain
backend, observability

Research direction

Read the prior feedback node rbuergi/Feedback/20260923-1655-logs-action-created-armed-lands-nothing and AccessContextPropagation.md, then inspect the Logs executor’s record-entries phase in the Hosting InstanceAction control plane (MeshWeaver.Plugins), especially the landing subscription’s CreateOrUpdateNodeRequest. Determine whether the triggering user’s AccessContext is propagated on the create-triggered route; the Loki timeout and 429 are separately described as likely environmental and need investigation. Done means the write-back identity defect is addressed and the separate Loki symptoms are assessed.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

The ProvisionPlan log-ingestion flow (Loki → Ops/Logs record phase, portal control plane) fails in two distinct ways at this log site. On the memex-cloud run, phase 4/4 read 47 log lines from Loki but 0 landed — every write-back was rejected with DeliveryFailureException: AccessContext must never be null for an application post, because the CreateOrUpdateNodeRequest posts carried no AccessContext (no Context, no CircuitContext). On the memex runs, phase 3/4 (the Loki query itself) failed: one 429 too many outstanding requests and two 45s budget timeouts against loki.monitoring.svc.cluster.local.

Probable cause

  • AccessContext loss (high confidence — known defect): the landing writes run on a subscription whose AccessContext is taken from the triggering change; on the create-triggered route it can be absent. This is documented in existing feedback (rbuergi/Feedback/20260923-1655-logs-action-created-armed-lands-nothing), which observed the identical symptom and notes the fix direction: capture the invoker's identity at trigger time via AccessContextScope.FromNode (per AGENTS.md / AccessContextPropagation.md), never rely on ambient context. Found while investigating Systemorph/MeshWeaver#5555 — this ticket is likely a production confirmation of that known issue, not a new defect.
  • Loki failures (lower confidence, environmental): the timeout and 429 appeared on the same 2-minute window across pods, which reads as Loki-side load/queue saturation rather than a client bug. The log's own text notes the timeout alone establishes the budget, not the cause. Cause unclear; evidence points at Loki capacity during that window.

Impact

Small: 4 occurrences over ~2 minutes on 3 portal pods, in the memex-cloud namespace. The run's selector and counts stay on the action node, so the window is recoverable by re-running — a broken secondary (ops/log-ingestion) path with an easy workaround, not a portal-facing outage.

Where to look

The Logs executor's record-entries phase in the Hosting InstanceAction control plane (MeshWeaver.Plugins), specifically where the landing subscription posts CreateOrUpdateNodeRequest without propagating the triggering user's AccessContext. The prior feedback node above has the concrete reproduction (armed-create vs re-arm) and the two candidate code paths.

Note for the filer: this log site fans out into at least three shapes (AccessContext write failure, Loki 45s timeout, Loki 429) that the masking rule currently folds into one fingerprint — the variants count says 1, which understates it. The AccessContext shape is a code defect; the Loki shapes are likely capacity symptoms and may warrant separate handling.

Evidence
Fingerprint fad474ac18061899
Category ProvisionPlan
Severity Error
Namespace memex-cloud
Pods memex-portal-deployment-76cf776847-2x9rq, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-76cf776847-5hflx
Occurrences 4
First seen 2026-09-24 06:52:39Z
Last seen 2026-09-24 06:54:42Z
Routing not determined — no configured route matches the category ProvisionPlan. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-24 06:52:39Z memex-portal-deployment-76cf776847-2x9rq fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex-cloud' FAILED in phase 4/4 'Record entries': the window was read (47 line(s)) and 0 of them landed in Ops/Logs; 47 could NOT be written — first fault: DeliveryFailureException: AccessContext must never be null for an application post — no identity, no delivery. hub=Ops/logs-memexcloud-20260924-verify-4859-j9vq8-b, message=CreateOrUpdateNodeRequest, target=portal/nodeops-MkvakB3S60a4rdhuPh7WLA was posted with no AccessContext (no Context, no CircuitContext). The post lost the user identity. Wire its source: user-context from the circuit/HTTP user; infrastructure via AccessService.ImpersonateAsSystem / PostOptions.ImpersonateAsHub; threads/activities via AccessContextScope.FromNode. See AccessContextPropagation.md / feedback_access_context_always_set.. The selector and both counts are on this node, so the window is not lost
2026-09-24 06:54:32Z memex-portal-deployment-76cf776847-v4txp fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 3/4 'Query the logs': http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/query_range did not answer within 45s — the read spent its whole budget and returned nothing. What that establishes is the budget, NOT the cause: it can be query volume, backend load, connectivity, or a response that started and stalled. Narrowing the window (sinceMinutes) or the streams (pod) is the cheapest next step, not a diagnosis — a 240-minute window over the LOW-volume namespace has spent this same budget, so volume is not established by a timeout alone
2026-09-24 06:54:32Z memex-portal-deployment-76cf776847-5hflx fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 3/4 'Query the logs': http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/query_range answered 429: too many outstanding requests
2026-09-24 06:54:42Z memex-portal-deployment-76cf776847-v4txp fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 3/4 'Query the logs': http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/query_range did not answer within 45s — the read spent its whole budget and returned nothing. What that establishes is the budget, NOT the cause: it can be query volume, backend load, connectivity, or a response that started and stalled. Narrowing the window (sinceMinutes) or the streams (pod) is the cheapest next step, not a diagnosis — a 240-minute window over the LOW-volume namespace has spent this same budget, so volume is not established by a timeout alone

Re-addressed by the current identity function: this incident inherited 05f47831e3b7d6cd (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/fad474ac18061899. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8aa2f9ea523cf69c: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 34m
Merged PRs (30d)
960

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.