Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Package store freezes the memex.meshweaver.cloud catalog while its manifest polls time out

Closed
  • Is something wrong?
#6,410 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
1-2 days
Newbie friendliness
14/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
csharp
Domain
backend, observability-sre

Research direction

Start at the StoreManifestSource poll code, searching the log category for the 'poll FAILED' and 'is DEGRADED' messages, and at the Polly timeout pipeline configured with the 90 s limit. The issue does not say what to change: first establish whether the timeouts come from the registry at memex.meshweaver.cloud or from the portal pod, which needs cluster-side access. Only then is there a concrete fix to propose.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

The portal's StoreManifestSource background poller cannot complete a manifest poll of the package registry at https://memex.meshweaver.cloud: three consecutive poll passes exceeded the pipeline's 90-second Polly timeout (TimeoutRejectedException wrapping a TaskCanceledException). While the source is DEGRADED, the store keeps serving that source's package list frozen at the last successful poll, so anything published to the registry in the meantime is missing from the catalog until a pass answers.

Probable cause

The registry endpoint is slow or not answering within the 90-second poll budget (high confidence — that is exactly what the timeout measures). Whether the slowness is registry-side (an outage of memex.meshweaver.cloud) or portal-side (network/pod distress) cannot be told from this single burst. Worth noting as context: the platform logged a silo-churn event with widespread SiloUnavailableException/TimeoutException incidents a few hours earlier the same night, so a lingering-unhealthy registry backend is plausible but unproven (low confidence).

Impact

Catalog staleness only, on one pod so far — the store itself stays up and serves the frozen list, and the poller self-heals on the first answering pass. This is a secondary-path degradation with a built-in workaround, hence sev:M; if further DEGRADED bursts for this source land, treat it as an outage of the registry and re-rule upward.

Where to look

  • Start at the StoreManifestSource poll code (log category StoreManifestSource; look for the sites emitting poll FAILED for <source> and <source> is DEGRADED (registry ...)) and the Polly retry + timeout pipeline configured with the 90 s timeout.
  • Independently, check the health/responsiveness of https://memex.meshweaver.cloud from inside the cluster — the poller's own degraded-state message already points there.
  • This is not a duplicate of the earlier StoreManifestSource incidents (Octokit RateLimitExceededException bursts and RegistryResponseException degraded states) — different failure signature and source kind.
Evidence
Fingerprint 4a60fecc09c51709
Category StoreManifestSource
Severity Error
Exception Polly.Timeout.TimeoutRejectedException
Top frame Polly.Retry.RetryResilienceStrategy1.ExecuteCore[TState](Func3 callback, ResilienceContext context, TState state)
Namespace memex-cloud
Pods memex-portal-deployment-7b55d9cd97-62jkm
Occurrences 1
First seen 2026-10-10 01:46:15Z
Last seen 2026-10-10 01:46:15Z
Routing not determined — no configured route matches the category StoreManifestSource. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-10 01:46:15Z memex-portal-deployment-7b55d9cd97-62jkm fail: StoreManifestSource[0]
      [StoreManifestSource] memex.meshweaver.cloud is DEGRADED (registry https://memex.meshweaver.cloud): ❌ 3 consecutive poll passes have failed (TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:01:30'.). The store is serving this source's package list FROZEN as of the last successful poll, 16 min before the latest failure — anything published since then is missing from the catalog. The fault looks transient, so the store refreshes by itself on the first pass that answers; if it does not clear, treat it as an outage of the source.
      Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:01:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread …[truncated]

Opened automatically from Admin/_LogIncident/4a60fecc09c51709. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 1cdb76794779e433: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
4h 15m
Merged PRs (30d)
969

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.