Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Node-operations hub drops the CreateNodeResponse for PackageHoldIntake's held-update ticket create — handler exits Processed, ticket never lands

Closed
  • Is something wrong?
#6,171 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Read the CreateNodeRequest handler on the portal/nodeops-* hub; the issue identifies its detached observable as the response path. Start by tracing its terminal arms and adding hub.NoteRequestStage(request.Id, …) so the next recurrence identifies the failed leg. The payload names no specific handler file or test; done means the response-loss path is diagnosable and the caller receives its CreateNodeResponse.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

The platform-build inbox watcher's package-hold intake on the memex control instance cannot record a held-update ticket (here: Mail 1.9.72 held on memex). PackageHoldIntake.Handle issues the CreateNodeRequest for the triage item under Hosting/Triage/package-update-held; the node-operations execution hub (portal/nodeops-*) enters its handler, records CREATE_CHAIN_SUBSCRIBED, exits Processed — and no CreateNodeResponse ever arrives, so the caller's 60 s timeout fires and the ticket is not recorded.

Probable cause

High confidence on the shape, low on the exact inner leg. The diagnostic trail rules out routing and a wedged target: every hop through the hosting, mesh-namespace and portal hubs is recorded with RunLevel=Started, and the handler exits Processed within ~17 ms — then no reply, completion or fault is recorded for the rest of the window. Canonical mesh handlers return Processed() at once and owe their reply from a detached observable, so the detached work behind the CreateNodeRequest handler either hung past the request window or terminated inside code that records no stage and dropped the reply.

Impact

One ticket lost on one pod in one window; a sibling incident of the same shape on the same hub (AppleIntelligence 1.1.23, Admin/_LogIncident/f7982eb301df2974) failed the same way minutes later. The instance re-sends the hold daily and the item is keyed by (deployment, package, held version, floor), so the retry is idempotent — this delays the ticket rather than losing it. A nuisance today, but it is the recurring node-ops reply-loss family and will keep surfacing wherever anything asks that hub to create, copy or move a node.

Where to look

This is the same entered-Processed-then-silence family on the node-operations hub as MeshWeaver#6170 (FleetTargetIntake SELF-UPDATE CreateNodeRequest, filed today) and MeshWeaver#6149 (PR sweep, same request type) — same hub, same request type, same shape; likely a duplicate, a human can close-as-dup into #6170 or #6149 in seconds. Also see #6105 (CopyNode), #6107 (MoveNode) and #2896 (per-node hub wedging). The caller is clean — its contract is exercised in PackageHoldIntakeTests. The defect is on the answering side: the CreateNodeRequest handler on the portal/nodeops-* hub whose detached observable owes the response. First step is the log's own suggestion: hub.NoteRequestStage(request.Id, …) at that handler's terminal arms, so the next recurrence names the leg that died.

Evidence
Fingerprint 35a91b88ffbfc4cf
Category PlatformBuildInboxWatcher
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-6cfcf78898-lsklh
Occurrences 1
First seen 2026-10-05 21:04:23Z
Last seen 2026-10-05 21:04:23Z
Routing not determined — no configured route matches the category PlatformBuildInboxWatcher. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 21:04:23Z memex-portal-deployment-6cfcf78898-lsklh fail: PlatformBuildInboxWatcher[0]
      [PackageHold] Mail 1.9.72 on memex could NOT be recorded — the instance re-sends daily
      System.TimeoutException: No response received in hub Hosting/PlatformBuilds within 00:01:00 for request CreateNodeRequest (id=ccvhlDKsIUWWbvjpDKjo9A) → target portal/nodeops-Qam49Nxrc06aKD_54VQZsg. This hub: RunLevel=Started Queue(buffer=0,deferred=0,openGates=0,drainsInFlight=0,draining=False,handledWhileWaiting=11). This hub is idle AT THE MOMENT IT GAVE UP — an instantaneous sample, which is why handledWhileWaiting above is printed beside it: that is the interval fact, and a hub that handled many messages was not idle throughout however empty its queue is now. Cause is UNKNOWN between: the target never received the request (routing), the target received it and is wedged (a per-node hub that stops answering — MeshWeaver#2896), or the target answered and the reply was lost. Queue state alone cannot distinguish them; the trail below can, and so can the target's own RunLevel and queue. Trail: AWAITING CreateNodeRequest→portal/nodeops-Qam49Nxrc06aKD_54VQZsg@Hosting/PlatformBuilds(+0ms) → POSTED target=portal/nodeops-Qam49Nxrc06aKD_54VQZsg@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@Hosting/PlatformBuilds(+0ms) → ENQUEUED@Hosting/PlatformBuilds(+0ms) → QUEUED queue=main depth=1@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@mesh/Qam49Nxrc06aKD_54VQZsg(+0ms) → ENQUEUED@mesh/Qam49Nxrc06aKD_54VQZsg(+0ms) → QUEUED queue=main depth=9@mesh/Qam49Nxrc06aKD_54VQZsg(+0ms) → ROUTED onTarget=False state=Forwarded@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@portal/nodeops-Qam49Nxrc06aKD_54VQZsg(+0ms) → ENQUEUED@portal/nodeops-Qam49Nxrc06aKD_54VQZsg(+0ms) → QUEUED queue=main depth=1@portal/nodeops-Qam49Nxrc06aKD_54VQZsg(+0ms) → ROUTED onTarget=False state=Forwarded@mesh/Qam49Nxrc06aKD_54VQZsg(+0ms) → ROUTED onTarget=True state=Submitted@portal/nodeops-Qam49Nxrc06aKD_54VQZsg(+17ms) → HANDLER_ENTER@portal/nodeops-Qam4…[truncated]

Opened automatically from Admin/_LogIncident/35a91b88ffbfc4cf. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 71b8bb715183c462: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 53m
Merged PRs (30d)
959

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.