Package store freezes the memex.meshweaver.cloud catalog while its manifest polls time out

Chiusa
  • Qualcosa non va?
#6,410 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
14/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend, observability-sre

Direzione di ricerca

Start at the StoreManifestSource poll code, searching the log category for the 'poll FAILED' and 'is DEGRADED' messages, and at the Polly timeout pipeline configured with the 90 s limit. The issue does not say what to change: first establish whether the timeouts come from the registry at memex.meshweaver.cloud or from the portal pod, which needs cluster-side access. Only then is there a concrete fix to propose.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:M

What is failing

The portal's StoreManifestSource background poller cannot complete a manifest poll of the package registry at https://memex.meshweaver.cloud: three consecutive poll passes exceeded the pipeline's 90-second Polly timeout (TimeoutRejectedException wrapping a TaskCanceledException). While the source is DEGRADED, the store keeps serving that source's package list frozen at the last successful poll, so anything published to the registry in the meantime is missing from the catalog until a pass answers.

Probable cause

The registry endpoint is slow or not answering within the 90-second poll budget (high confidence — that is exactly what the timeout measures). Whether the slowness is registry-side (an outage of memex.meshweaver.cloud) or portal-side (network/pod distress) cannot be told from this single burst. Worth noting as context: the platform logged a silo-churn event with widespread SiloUnavailableException/TimeoutException incidents a few hours earlier the same night, so a lingering-unhealthy registry backend is plausible but unproven (low confidence).

Impact

Catalog staleness only, on one pod so far — the store itself stays up and serves the frozen list, and the poller self-heals on the first answering pass. This is a secondary-path degradation with a built-in workaround, hence sev:M; if further DEGRADED bursts for this source land, treat it as an outage of the registry and re-rule upward.

Where to look

  • Start at the StoreManifestSource poll code (log category StoreManifestSource; look for the sites emitting poll FAILED for <source> and <source> is DEGRADED (registry ...)) and the Polly retry + timeout pipeline configured with the 90 s timeout.
  • Independently, check the health/responsiveness of https://memex.meshweaver.cloud from inside the cluster — the poller's own degraded-state message already points there.
  • This is not a duplicate of the earlier StoreManifestSource incidents (Octokit RateLimitExceededException bursts and RegistryResponseException degraded states) — different failure signature and source kind.
Evidence
Fingerprint 4a60fecc09c51709
Category StoreManifestSource
Severity Error
Exception Polly.Timeout.TimeoutRejectedException
Top frame Polly.Retry.RetryResilienceStrategy1.ExecuteCore[TState](Func3 callback, ResilienceContext context, TState state)
Namespace memex-cloud
Pods memex-portal-deployment-7b55d9cd97-62jkm
Occurrences 1
First seen 2026-10-10 01:46:15Z
Last seen 2026-10-10 01:46:15Z
Routing not determined — no configured route matches the category StoreManifestSource. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-10 01:46:15Z memex-portal-deployment-7b55d9cd97-62jkm fail: StoreManifestSource[0]
      [StoreManifestSource] memex.meshweaver.cloud is DEGRADED (registry https://memex.meshweaver.cloud): ❌ 3 consecutive poll passes have failed (TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:01:30'.). The store is serving this source's package list FROZEN as of the last successful poll, 16 min before the latest failure — anything published since then is missing from the catalog. The fault looks transient, so the store refreshes by itself on the first pass that answers; if it does not clear, treat it as an outage of the source.
      Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:01:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread …[truncated]

Opened automatically from Admin/_LogIncident/4a60fecc09c51709. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 1cdb76794779e433: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 10m
PR unite (30g)
980

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.