Webhook inbox delivery answers a bare 500 on a fan-in stall instead of 503, and is starved by the known pedestrian-walk pg-read head-of-line blocking

Open

#6,125 opened on Oct 5, 2026 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty: 4/5
Estimated time: 3-5 days
Newbie friendliness: 45/100

Issue type: Bug
Clarity: Mostly clear
Activity status: Active
Tech stack: csharp, postgresql
Domain: api, backend, databases

Research direction

Read Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs at the Deliver route and compare its stalled-read handling with PluginBundleEndpoints and #5345. Check whether the documented provider walk fix has rolled out, consulting Doc/Architecture/QueryFanInStallTerminal and ThreadSupervisor.Scan; do not change the query budget. Done means stalled reads return the instance-key 503 with Retry-After, while unrelated faults still surface, with coverage for the response behavior.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

The webhook inbox delivery route (WebhookInboxEndpoints.Deliver, Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:108) faults when its path:Hosting/PlatformBuilds read misses the query fan-in's 15 s Initial budget — a one-row exact probe issued under the system identity. The probe itself is not the defect: it is the known victim of the pedestrian satellite walk's head-of-line starvation of the process-wide pg-read:Postgres pool (cap 16), documented 2026-10-04 in Doc/Architecture/QueryFanInStallTerminal → "Head-of-line starvation of pg-read by the pedestrian satellite walk". A second, endpoint-local defect: the route lets QueryProviderStalledException escape as an unhandled exception, so ASP.NET answers a bare 500 instead of the instance-key 503 + Retry-After convention the plugin-bundle routes adopted after #5345 (InstanceAuthResponses.UnavailableOnAStalledRead, classifier AreaErrorClassifier.IsStorageUnavailable).

Probable cause

High confidence on the mechanism, because the architecture page records this exact query shape and provider as one of the two recurring stall shapes between 2026-09-27 and 2026-10-04: ThreadSupervisor.Scan's newest/Admin thread query drove StorageAdapterMeshQueryProvider's unbounded pedestrian walk (~2×N pg-read admissions, unbounded SelectMany), and every read issued after it — including the one-row Hosting/PlatformBuilds probe — waited behind the walk's whole remainder. A two-part fix (defer scoped satellite reads to the native provider; bound the walk to WalkConcurrency = 4, pinned by PedestrianWalkHeadOfLineTest) is recorded as made but its rollout to the memex portal pods is not established; today's recurrence suggests these pods do not yet carry it, or the walk is not the whole cause (a per-query pool-admission census still does not exist). The endpoint's bare-500 mapping is a separate, certain defect: the fan-in's terminal is doing exactly what it is for, and this consumer — unlike the plugin-bundle routes — was never converted.

Impact

Small: a handful of occurrences across two portal pods over ~12 hours, so an intermittent 500 on webhook delivery, retryable by the sender. The stall itself is a fleet-wide availability risk (routing gates on the same fan-in) tracked under #5315; this ticket is about this endpoint's share of the blast radius.

Where to look

Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs (the Deliver route) — map QueryProviderStalledException to 503 + Retry-After via InstanceAuthResponses.UnavailableOnAStalledRead, as PluginBundleEndpoints does; any other fault still escapes so real defects surface.
MeshWeaver.Hosting.StorageAdapterMeshQueryProvider — confirm the 2026-10-04 walk fix (defer scoped satellite reads, WalkConcurrency bound) has actually rolled to the memex portal images; the doc pins the mechanism but records no production timing data.
MeshWeaver.Plugins ThreadSupervisor.Scan — the sweep that issues the Admin/Thread walk.
Context: Doc/Architecture/QueryFanInStallTerminal, issue #5345 (the same escape-as-500 defect in a sibling route), and #5315 (the open provider-stall investigation this instance belongs to). Do not bump QueryInitialBudget; policy query-fanin-stall-terminal is fixed and correct here.

Evidence


Fingerprint	`b1942b90104d63ed`
Category	`Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware`
Severity	Error
Exception	`MeshWeaver.Mesh.QueryProviderStalledException`
Top frame	`Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct)`
Namespace	`memex`
Pods	`memex-portal-deployment-76dd69c7fc-xxql9`, `memex-portal-deployment-6d8c9665cc-jltnc`
Occurrences	3
First seen	2026-10-04 19:25:46Z
Last seen	2026-10-05 07:44:45Z
Routing	not determined — no configured route matches the category `Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-10-04 19:25:46Z memex-portal-deployment-76dd69c7fc-xxql9 fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Hosting/PlatformBuilds' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
         at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 108
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
         at Memex.Portal.Shared.Authentication.OnboardingMiddleware.Continue(HttpContext context, IObservable`1 pendingDecision) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 191
         at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
         at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 319
         at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass12_0.<<UsePublicHostRedirect>b__0>d.MoveNext() in /home/runner/work/MeshWeav…[truncated]
2026-10-04 19:25:46Z memex-portal-deployment-76dd69c7fc-xxql9 fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNP27FHC82EM", Request id "0HNP27FHC82EM:00000001": An unhandled exception was thrown by the application.
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Hosting/PlatformBuilds' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
         at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 108
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
         at Memex.Portal.Shared.Authentication.OnboardingMiddleware.Continue(HttpContext context, IObservable`1 pendingDecision) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 191
         at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
         at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 319
         at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass12_0.<<UsePublicHostRedirect>b__0>d.MoveNe…[truncated]
2026-10-05 07:44:45Z memex-portal-deployment-6d8c9665cc-jltnc fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Hosting/PlatformBuilds' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
         at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 108
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
         at Memex.Portal.Shared.Authentication.OnboardingMiddleware.Continue(HttpContext context, IObservable`1 pendingDecision) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 191
         at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
         at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 319
         at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass12_0.<<UsePublicHostRedirect>b__0>d.MoveNext() in /home/runner/work/MeshWeav…[truncated]

_{Opened automatically from Admin/_LogIncident/b1942b90104d63ed. Recurrences are folded into this issue rather than opening new ones.}
_{It also stands for the whole log site 4660764b5f543068: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}

Dominant language: C#
Stars: 12
Forks: 5
Avg merge: 3h 53m
Merged PRs (30d): 968

Getting set up

No Dockerfile or Docker Compose file
Has a pull request template
No contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

Post-merge review of PR #6210 (head 70391cb): PassOver and GuardedPass XML doc openers read as stray </summary> in the stored diff - sev:L, needs one look at the file
Open
- Is something wrong?
sev:L

Difficulty 1/5 Under an hour Newbie friendliness 72/100

Systemorph/MeshWeaver#6233 · Opened Oct 7, 2026

Maintainers usually reply within 1 day
Doc/AI/ModelProviderSettings references removed ModelsSettingsTab.cs — update to point at Providers/ProvidersApp (ProviderSetup, /Provider/AiModels)
Open
- Is something wrong?
documentation feedback sev:L

Difficulty 1/5 Under an hour Newbie friendliness 82/100

Systemorph/MeshWeaver#6033 · Opened Oct 3, 2026

Maintainers usually reply within 1 day
Setup wizard and memex-local say search "falls back to matching words" without embeddings — on PostgreSQL, words in a node's content are not searched at all
Open
- Is something wrong?
area:search documentation

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

Systemorph/MeshWeaver#6030 · Opened Oct 3, 2026

Maintainers usually reply within 1 day
ApiTokenService.RevokeToken posts its revocation SaveMeshNodeRequest from the mesh (router) hub instead of a node-operation hub
Open
- Is something wrong?
bug sev:M

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

Systemorph/MeshWeaver#6026 · 1 comment · Opened Oct 3, 2026

Maintainers usually reply within 1 day
memex-local verify reports loaded view packs as NOT installed: the on-disk probe does not search /app/modules
Open
- Is something wrong?
area:hosting bug sev:L

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

Systemorph/MeshWeaver#6025 · Opened Oct 3, 2026

Maintainers usually reply within 1 day

All issues in Systemorph/MeshWeaver

Similar issues

Merged cabinet extraction leaves consumed cabinet files behind
Open
- Is something wrong?
Difficulty 2/5 1-3 hours Newbie friendliness 82/100

activescott/lessmsi#306 · Opened Oct 7, 2026
Python: Bug: split_plaintext_paragraph / split_markdown_paragraph can return a chunk larger than max_tokens
Possibly taken @xThreeh claimed this today. Open
- Is something wrong?
python triage

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

microsoft/semantic-kernel#14566 · Opened Oct 7, 2026

Maintainers usually reply within 4 days
CI: perf Pester step passes when a test file fails to load
Open
- Is something wrong?
triage

Difficulty 1/5 Under an hour Newbie friendliness 82/100

rjmurillo/moq.analyzers#1384 · Opened Oct 7, 2026
Variables passed to Compensated are not set on the routing slip
May be free again A pull request for this issue was closed without being merged. Open
- Is something wrong?
Difficulty 2/5 1-3 hours Newbie friendliness 75/100

MassTransit/MassTransit#6249 · Opened Oct 7, 2026
Artwork bytes are handed to the application as 'JPEG/PNG' without any check against the channel's declared format
Open
- Is something wrong?
security

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

Sendspin/sendspin-dotnet#339 · Opened Oct 7, 2026

Maintainers usually reply within 1 day

More C# issues

Webhook inbox delivery answers a bare 500 on a fan-in stall instead of 503, and is starved by the known pedestrian-walk pg-read head-of-line blocking

Is something wrong?

Assessment

Research direction

Description

Evidence

Getting set up

First steps

More from Systemorph/MeshWeaver

Is something wrong?

Is something wrong?

Is something wrong?

Is something wrong?

Is something wrong?

Similar issues

Is something wrong?

Is something wrong?

Is something wrong?

Is something wrong?

Is something wrong?

Get new issues in your inbox