Webhook endpoint answers after post-creation handlers: GitHub org-hook delivery recorded 500 (10 s timeout) although the event was stored (roll window)

Chiusa
  • Qualcosa non va?
#6,039 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
api, backend

Direzione di ricerca

Start with memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs and the MeshExtensions CreateNode post-creation path. Trace when the inbox store commits, when post-creation handlers and response routing run, and when the endpoint sends its response; the report does not establish which stage stalls. Done means the webhook can receive a successful response after storage commits without waiting on post-creation work, while preserving event processing.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:hosting bug sev:M

Moved from the fleet's private triage inbox (MeshWeaver.Feedback#47, filed 2026-09-26). Core owns it: memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs is core.

Owner: core Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs (+ MeshExtensions CreateNode post-creation path). Measured 2026-09-26 on memex.systemorph.com. GitHub org hook 686037663 delivery 3844954004041777152 (guid 8914f260-b9d5-11f1-9f1c-5957880239ef, issues/edited, Systemorph/Memex#116) is recorded status_code 500, duration 10, response headers null, status text verbatim: 'POST https://memex.systemorph.com/api/hooks/Hosting/PlatformBuilds giving up after 1 attempt(s): ... context deadline exceeded (Client.Timeout exceeded while awaiting headers)'. So the 500 is GitHub's own label for NO RESPONSE in 10 s, not a server 500. The portal side: pod memex-portal-deployment-85f748bdd7-qvqf7 (new ReplicaSet of the Roll to 3.0.0-ci.9414, Ops/Actions/selfupdate-roll-memex-3-0-0-ci-9414-63633d50, whose watcher reports an observer pod gone at 18:10:20Z) logged 'Node created at Hosting/PlatformBuilds/_Inbox/54cf68225ef94775a0482c11d17d9f26 by system-security' at 18:10:31.369Z, i.e. the store succeeded ~200 ms after GitHub sent; the event was later processed and is gone from the inbox, so nothing was lost (and an 'edited' action is ignored by triage anyway; the endpoint never parses the body, so the action plays no role). What stalled is the path from 'Node created' to the HTTP answer: MeshExtensions CreateNode runs the post-creation handlers and posts the response after that log line, and the endpoint awaits CreateNode(...).Take(1) before answering. The last 100 deliveries: p50 0.46 s, p90 2.38 s, max 10 s; the 13 after 18:13Z (still in the roll) took 1.0-4.9 s. Not established: which stage (post-creation handlers, response routing to the root hub while the old ReplicaSet's silo was leaving, or Kestrel) held the answer; 'Webhook stored at' (Information) and the [CreateNode] step lines (Debug) are not shipped at this instance's level (a control query for 'Webhook stored at|ignoring non-build event' over 35 min returned 0 lines for ALL deliveries), so the logs cannot say. Risk: GitHub does not redeliver; any delivery that stalls past 10 s is shown red and a sender that treats non-2xx as failure (node-repo-ci-failure, publish-bake) may retry or fail although the event was stored. Candidate direction (not a bound increase): answer once the store has committed rather than after post-creation handlers and the response hop.

Context (added by triage):

  • Category: bug (verified defect report, agent session, Claude Code)
  • Where: no specific Space/portal node named — observed during the memex.systemorph.com self-update roll to 3.0.0-ci.9414; affects the portal webhook inbox path generally.
  • Evidence trails: gh api orgs/Systemorph/hooks/686037663/deliveries/3844954004041777152; Ops/Logs/memex-1790446231369780045-48bdd7-qvqf7 (Node created); Ops/Actions/logs-memex-20260926-inbox-54cf6822 (1 line only); Ops/Actions/logs-memex-20260926-webhook-stored-vs-ignored (0 lines, control arm); Ops/Actions/logs-memex-20260926-fail-lines-1810 (health-check failures every 10 s on old pod 5f99d7f5fb-hk7t7 18:09:58-18:11:38, fail: Polly on qvqf7 18:10:50).
  • Origin: https://memex.systemorph.com/rbuergi/Feedback/org-webhook-timeout-during-roll-20260926

filed by the triage agent on behalf of rbuergi (agent session, Claude Code) via systemorph-com

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 11m
PR unite (30g)
978

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.