Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Webhook endpoint answers after post-creation handlers: GitHub org-hook delivery recorded 500 (10 s timeout) although the event was stored (roll window)

Closed
  • Is something wrong?
#6,039 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
api, backend

Research direction

Start with memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs and the MeshExtensions CreateNode post-creation path. Trace when the inbox store commits, when post-creation handlers and response routing run, and when the endpoint sends its response; the report does not establish which stage stalls. Done means the webhook can receive a successful response after storage commits without waiting on post-creation work, while preserving event processing.

Written by the indexing model from the issue text.

Description

area:hosting bug sev:M

Moved from the fleet's private triage inbox (MeshWeaver.Feedback#47, filed 2026-09-26). Core owns it: memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs is core.

Owner: core Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs (+ MeshExtensions CreateNode post-creation path). Measured 2026-09-26 on memex.systemorph.com. GitHub org hook 686037663 delivery 3844954004041777152 (guid 8914f260-b9d5-11f1-9f1c-5957880239ef, issues/edited, Systemorph/Memex#116) is recorded status_code 500, duration 10, response headers null, status text verbatim: 'POST https://memex.systemorph.com/api/hooks/Hosting/PlatformBuilds giving up after 1 attempt(s): ... context deadline exceeded (Client.Timeout exceeded while awaiting headers)'. So the 500 is GitHub's own label for NO RESPONSE in 10 s, not a server 500. The portal side: pod memex-portal-deployment-85f748bdd7-qvqf7 (new ReplicaSet of the Roll to 3.0.0-ci.9414, Ops/Actions/selfupdate-roll-memex-3-0-0-ci-9414-63633d50, whose watcher reports an observer pod gone at 18:10:20Z) logged 'Node created at Hosting/PlatformBuilds/_Inbox/54cf68225ef94775a0482c11d17d9f26 by system-security' at 18:10:31.369Z, i.e. the store succeeded ~200 ms after GitHub sent; the event was later processed and is gone from the inbox, so nothing was lost (and an 'edited' action is ignored by triage anyway; the endpoint never parses the body, so the action plays no role). What stalled is the path from 'Node created' to the HTTP answer: MeshExtensions CreateNode runs the post-creation handlers and posts the response after that log line, and the endpoint awaits CreateNode(...).Take(1) before answering. The last 100 deliveries: p50 0.46 s, p90 2.38 s, max 10 s; the 13 after 18:13Z (still in the roll) took 1.0-4.9 s. Not established: which stage (post-creation handlers, response routing to the root hub while the old ReplicaSet's silo was leaving, or Kestrel) held the answer; 'Webhook stored at' (Information) and the [CreateNode] step lines (Debug) are not shipped at this instance's level (a control query for 'Webhook stored at|ignoring non-build event' over 35 min returned 0 lines for ALL deliveries), so the logs cannot say. Risk: GitHub does not redeliver; any delivery that stalls past 10 s is shown red and a sender that treats non-2xx as failure (node-repo-ci-failure, publish-bake) may retry or fail although the event was stored. Candidate direction (not a bound increase): answer once the store has committed rather than after post-creation handlers and the response hop.

Context (added by triage):

  • Category: bug (verified defect report, agent session, Claude Code)
  • Where: no specific Space/portal node named — observed during the memex.systemorph.com self-update roll to 3.0.0-ci.9414; affects the portal webhook inbox path generally.
  • Evidence trails: gh api orgs/Systemorph/hooks/686037663/deliveries/3844954004041777152; Ops/Logs/memex-1790446231369780045-48bdd7-qvqf7 (Node created); Ops/Actions/logs-memex-20260926-inbox-54cf6822 (1 line only); Ops/Actions/logs-memex-20260926-webhook-stored-vs-ignored (0 lines, control arm); Ops/Actions/logs-memex-20260926-fail-lines-1810 (health-check failures every 10 s on old pod 5f99d7f5fb-hk7t7 18:09:58-18:11:38, fail: Polly on qvqf7 18:10:50).
  • Origin: https://memex.systemorph.com/rbuergi/Feedback/org-webhook-timeout-during-roll-20260926

filed by the triage agent on behalf of rbuergi (agent session, Claude Code) via systemorph-com

Dominant language
C#
Stars
12
Forks
5
Avg merge
4h 24m
Merged PRs (30d)
961

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.