Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

CreateNodeRequest handler on the node-operations hub exits Processed without replying — PR sweep review-thread kick times out after 60 s

Open
  • Is something wrong?
#6,148 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Start at Hosting/Deployment/Source/PullRequestSweep and trace Kick to PullRequestIntake.Handle to verify the caller, then inspect the CreateNodeRequest handler registered on the node-operations execution hub. Read related issues #6105 and #6107; instrument the handler’s terminal arms with NoteRequestStage and use the next recurrence to identify the stalled leg. Done means the handler’s detached work completes with a CreateNodeResponse and the kick no longer times out.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

The pull-request sweep's kick path feeds an unreviewed head through PullRequestIntake.Handle, which issues a CreateNodeRequest to the portal's node-operations execution hub to stand up the review-thread/item node for the PR. The request reached the target hub and was routed on-target — a handler was entered, subscribed the create chain, and exited with state Processed within ~20 ms — but no CreateNodeResponse ever arrived, so the caller's 60 s request timeout fired and the kick was recorded as a run failure ([PrSweep] kicking Systemorph/MeshWeaver.Plugins#2904 … failed). The caller side behaved as designed: the sweep caught the timeout, named the failure on the run, and left the head to be re-decided next run.

Probable cause

Moderate confidence on the shape, low on the exact spot. The diagnostic trail rules out routing: every hop is recorded (AWAITING → POSTED → RECEIVED → ENQUEUED → QUEUED → ROUTED onTarget=True state=Submitted → HANDLER_ENTER → CREATE_CHAIN_SUBSCRIBED → HANDLER_EXIT state=Processed), and after the handler exit no reply, completion or fault was recorded for the whole 60 s window. Canonical mesh handlers return Processed() at once and owe their reply from a detached observable, so the detached work behind the CreateNodeRequest handler either hung (still running past the request window) or terminated inside code that records no stage and dropped the reply. The inner stages of that handler are not instrumented, so this evidence cannot say which. This is the third member of the same family on this hub: MeshWeaver#6105 (CopyNode's detached reply chain outlives the 60 s RequestTimeout) and #6107 (MoveNodeRequest handler exits Processed and never replies) — each a different request type, the same entered-Processed-then-silence shape on the node-operations hub. The log's own suggested fix is the right first step: call hub.NoteRequestStage(request.Id, …) at the create handler's terminal arms so the next recurrence names the leg that died.

Impact

One kick of one head (MeshWeaver.Plugins#2904) on one pod was lost. The consequence is a delayed review, not a lost one: the head still carries no internal-review check run and no recorded review thread, so the next sweep run (default every 10 minutes) decides it Kick again. Judged from the numbers (1 occurrence, 1 pod, single window) this is a nuisance today — but the same reply-loss family has now struck the node-ops hub three times in two days across three callers (feedback relocation, PR sweep kicks, and the CopyNode path), so the underlying defect is live and will keep surfacing wherever the hub is asked to create or move a node.

Where to look

Start at Hosting/Deployment/Source/PullRequestSweep (Kick → PullRequestIntake.Handle) to confirm the caller is clean — it is; the defect is on the answering side: the CreateNodeRequest handler registered on the node-operations execution hub (portal/nodeops-*), whose detached observable owes the CreateNodeResponse. Instrument that handler's terminal arms with NoteRequestStage and re-run; the next occurrence will name the stuck leg. Related prior art worth reading before touching it: MeshWeaver#6105 and #6107 (same hub, same shape), and MeshWeaver#2896 for the per-node hub wedging case this could turn out to be.

Evidence
Fingerprint 6ee25e8183c8c62a
Category PullRequestSweep
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-848f664fd5-zqfrh
Occurrences 1
First seen 2026-10-05 10:06:19Z
Last seen 2026-10-05 10:06:19Z
Routing not determined — no configured route matches the category PullRequestSweep. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 10:06:19Z memex-portal-deployment-848f664fd5-zqfrh fail: PullRequestSweep[0]
      [PrSweep] kicking Systemorph/MeshWeaver.Plugins#2904 at 60ec61069def05260451f293a88967bd45136162 failed
      System.TimeoutException: No response received in hub Hosting/PlatformBuilds within 00:01:00 for request CreateNodeRequest (id=fz1QIYHoT06wip2IB2aQIw) → target portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg. This hub: RunLevel=Started Queue(buffer=0,deferred=0,openGates=0,drainsInFlight=0,draining=False,handledWhileWaiting=4). This hub is idle AT THE MOMENT IT GAVE UP — an instantaneous sample, which is why handledWhileWaiting above is printed beside it: that is the interval fact, and a hub that handled many messages was not idle throughout however empty its queue is now. Cause is UNKNOWN between: the target never received the request (routing), the target received it and is wedged (a per-node hub that stops answering — MeshWeaver#2896), or the target answered and the reply was lost. Queue state alone cannot distinguish them; the trail below can, and so can the target's own RunLevel and queue. Trail: AWAITING CreateNodeRequest→portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg@Hosting/PlatformBuilds(+0ms) → POSTED target=portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@Hosting/PlatformBuilds(+0ms) → ENQUEUED@Hosting/PlatformBuilds(+0ms) → QUEUED queue=main depth=1@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+0ms) → ENQUEUED@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+0ms) → QUEUED queue=main depth=1@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+0ms) → ROUTED onTarget=False state=Forwarded@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → ENQUEUED@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → QUEUED queue=main depth=1@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → ROUTED onTarget=False state=Forwarded@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → ROUTED onTarget=True state=Submitted@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+20ms) → HANDLER_ENTER@portal/nodeo…[truncated]

Opened automatically from Admin/_LogIncident/6ee25e8183c8c62a. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 11a37046d2a96572: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
3h 53m
Merged PRs (30d)
968

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.