CreateNodeRequest handler on the node-operations hub exits Processed without replying — PR sweep review-thread kick times out after 60 s

Chiusa
  • Qualcosa non va?
#6,148 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend

Direzione di ricerca

Start at Hosting/Deployment/Source/PullRequestSweep and trace Kick to PullRequestIntake.Handle to verify the caller, then inspect the CreateNodeRequest handler registered on the node-operations execution hub. Read related issues #6105 and #6107; instrument the handler’s terminal arms with NoteRequestStage and use the next recurrence to identify the stalled leg. Done means the handler’s detached work completes with a CreateNodeResponse and the kick no longer times out.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:M

What is failing

The pull-request sweep's kick path feeds an unreviewed head through PullRequestIntake.Handle, which issues a CreateNodeRequest to the portal's node-operations execution hub to stand up the review-thread/item node for the PR. The request reached the target hub and was routed on-target — a handler was entered, subscribed the create chain, and exited with state Processed within ~20 ms — but no CreateNodeResponse ever arrived, so the caller's 60 s request timeout fired and the kick was recorded as a run failure ([PrSweep] kicking Systemorph/MeshWeaver.Plugins#2904 … failed). The caller side behaved as designed: the sweep caught the timeout, named the failure on the run, and left the head to be re-decided next run.

Probable cause

Moderate confidence on the shape, low on the exact spot. The diagnostic trail rules out routing: every hop is recorded (AWAITING → POSTED → RECEIVED → ENQUEUED → QUEUED → ROUTED onTarget=True state=Submitted → HANDLER_ENTER → CREATE_CHAIN_SUBSCRIBED → HANDLER_EXIT state=Processed), and after the handler exit no reply, completion or fault was recorded for the whole 60 s window. Canonical mesh handlers return Processed() at once and owe their reply from a detached observable, so the detached work behind the CreateNodeRequest handler either hung (still running past the request window) or terminated inside code that records no stage and dropped the reply. The inner stages of that handler are not instrumented, so this evidence cannot say which. This is the third member of the same family on this hub: MeshWeaver#6105 (CopyNode's detached reply chain outlives the 60 s RequestTimeout) and #6107 (MoveNodeRequest handler exits Processed and never replies) — each a different request type, the same entered-Processed-then-silence shape on the node-operations hub. The log's own suggested fix is the right first step: call hub.NoteRequestStage(request.Id, …) at the create handler's terminal arms so the next recurrence names the leg that died.

Impact

One kick of one head (MeshWeaver.Plugins#2904) on one pod was lost. The consequence is a delayed review, not a lost one: the head still carries no internal-review check run and no recorded review thread, so the next sweep run (default every 10 minutes) decides it Kick again. Judged from the numbers (1 occurrence, 1 pod, single window) this is a nuisance today — but the same reply-loss family has now struck the node-ops hub three times in two days across three callers (feedback relocation, PR sweep kicks, and the CopyNode path), so the underlying defect is live and will keep surfacing wherever the hub is asked to create or move a node.

Where to look

Start at Hosting/Deployment/Source/PullRequestSweep (Kick → PullRequestIntake.Handle) to confirm the caller is clean — it is; the defect is on the answering side: the CreateNodeRequest handler registered on the node-operations execution hub (portal/nodeops-*), whose detached observable owes the CreateNodeResponse. Instrument that handler's terminal arms with NoteRequestStage and re-run; the next occurrence will name the stuck leg. Related prior art worth reading before touching it: MeshWeaver#6105 and #6107 (same hub, same shape), and MeshWeaver#2896 for the per-node hub wedging case this could turn out to be.

Evidence
Fingerprint 6ee25e8183c8c62a
Category PullRequestSweep
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-848f664fd5-zqfrh
Occurrences 1
First seen 2026-10-05 10:06:19Z
Last seen 2026-10-05 10:06:19Z
Routing not determined — no configured route matches the category PullRequestSweep. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 10:06:19Z memex-portal-deployment-848f664fd5-zqfrh fail: PullRequestSweep[0]
      [PrSweep] kicking Systemorph/MeshWeaver.Plugins#2904 at 60ec61069def05260451f293a88967bd45136162 failed
      System.TimeoutException: No response received in hub Hosting/PlatformBuilds within 00:01:00 for request CreateNodeRequest (id=fz1QIYHoT06wip2IB2aQIw) → target portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg. This hub: RunLevel=Started Queue(buffer=0,deferred=0,openGates=0,drainsInFlight=0,draining=False,handledWhileWaiting=4). This hub is idle AT THE MOMENT IT GAVE UP — an instantaneous sample, which is why handledWhileWaiting above is printed beside it: that is the interval fact, and a hub that handled many messages was not idle throughout however empty its queue is now. Cause is UNKNOWN between: the target never received the request (routing), the target received it and is wedged (a per-node hub that stops answering — MeshWeaver#2896), or the target answered and the reply was lost. Queue state alone cannot distinguish them; the trail below can, and so can the target's own RunLevel and queue. Trail: AWAITING CreateNodeRequest→portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg@Hosting/PlatformBuilds(+0ms) → POSTED target=portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@Hosting/PlatformBuilds(+0ms) → ENQUEUED@Hosting/PlatformBuilds(+0ms) → QUEUED queue=main depth=1@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+0ms) → ENQUEUED@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+0ms) → QUEUED queue=main depth=1@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+0ms) → ROUTED onTarget=False state=Forwarded@Hosting/PlatformBuilds(+0ms) → RECEIVED runLevel=Started@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → ENQUEUED@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → QUEUED queue=main depth=1@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → ROUTED onTarget=False state=Forwarded@mesh/jy8tTG7Ud0S8g1qi28Z5Tg(+1ms) → ROUTED onTarget=True state=Submitted@portal/nodeops-jy8tTG7Ud0S8g1qi28Z5Tg(+20ms) → HANDLER_ENTER@portal/nodeo…[truncated]

Opened automatically from Admin/_LogIncident/6ee25e8183c8c62a. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 11a37046d2a96572: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 8m
PR unite (30g)
979

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.