Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

mw-log-watcher stopped polling memex for ~9 minutes: one pass read a window far wider than the poll interval with an empty delivery queue

Aperta
  • Qualcosa non va?
#7,043 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp, grafana, kubernetes
Ambito
backend, observability

Direzione di ricerca

Start with LogWatchWorker and WatcherState.CursorFor in src/MeshWeaver.LogWatcher, then read the "Delivery guarantees" and "The watcher tickets its own blind spots" sections of LogWatchTriage. Check the watcher pod logs with the provided kubectl command across a lag window to distinguish a restart from a stalled poll loop. Done means the cause is established and the watcher avoids repeated polling gaps; verify behavior against the cursor and MaxCatchUp contract.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:M

What is failing

The mw-log-watcher serving namespace memex went roughly nine poll intervals without completing a pass: its next Loki query_range covered a window far wider than the 1-minute PollInterval, and the store answered it — so nothing was lost, but nothing in that stretch was read, grouped, queued or delivered until the late pass ran, and every red log in it was ticketed that late. This is the watcher ticketing its own blind spot (the Pipeline behind-finding), and the delay directly corrupts the incident record: an incident's createdDate is its delivery time, so a lag this size makes the record read as "ticketing stopped".

Probable cause

The message offers two candidates — the watcher wasn't polling, or a delivery backlog held the pass. The sample rules the second one out: zero reports were still undelivered when the pass began, so the delivery queue was empty and delivery cannot have blocked the poll loop. That leaves the first: the watcher was not polling for that stretch. Most likely a pod restart or a stalled/blocked poll loop (a hung Loki query, scheduling starvation, or a crash-loop window on the monitoring Deployment). Confidence: moderate — a single occurrence with one sample and no pod list on the incident; the empty-queue reading is solid, but distinguishing restart from stall needs the watcher's own pod log (kubectl -n monitoring logs deploy/mw-log-watcher), which is not readable from the portal.

Impact

Single occurrence, single namespace, no data lost — the store answered the wide window. This is a delayed-detection defect, not an outage. But it is the precursor of the condition that does lose data: a lag that keeps growing ends at the MaxCatchUp floor, where the skipped window is never read. That exact endpoint already happened on the sibling namespace — see Admin/_LogIncident/log-window-skipped-memex-cloud (filed as MeshWeaver.Plugins#2156, superseded by MeshWeaver#4326). Worth watching whether this recurs with a growing window rather than a one-off restart blip.

Where to look

  1. The mw-log-watcher Deployment (namespace monitoring) serving memex — its pod log across the lag window, to tell restart-crash from loop-stall.
  2. The poll loop and cursor advance in src/MeshWeaver.LogWatcher (MeshWeaver.Plugins): LogWatchWorker, WatcherState.CursorFor — check whether anything between "advance cursor" and the next poll can block (a slow query_range with no timeout, or a synchronous step inside the loop).
  3. LogWatchTriage — sections "Delivery guarantees" and "The watcher tickets its own blind spots" describe the cursor/MaxCatchUp contract this finding guards.

Already known?

Not a duplicate, but the same log site as two filed siblings: log-burst-header-only-memex and log-window-skipped-memex-cloud (site fold df7022d82cca1b39, MeshWeaver.LogWatcher.Pipeline). Those are different findings (bodyless captures; a MaxCatchUp-floored skip); this one is the behind-but-recoverable predecessor of the latter.

Evidence
Fingerprint log-pipeline-behind-memex
Category MeshWeaver.LogWatcher.Pipeline
Severity Error
Namespace memex
Occurrences 4
First seen 2026-10-05 10:29:44Z
Last seen 2026-10-21 17:58:20Z
Routing not determined — no configured route matches the category MeshWeaver.LogWatcher.Pipeline. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 10:38:33Z - Loki query_range [2026-10-05 10:29:44Z, 2026-10-05 10:38:33Z) — 9 minute(s) behind, 0 report(s) still undelivered when the pass began
2026-10-05 10:41:48Z - Loki query_range [2026-10-05 10:35:21Z, 2026-10-05 10:41:48Z) — 6 minute(s) behind, 89 report(s) still undelivered when the pass began
2026-10-05 21:02:31Z - Loki query_range [2026-10-05 20:52:51Z, 2026-10-05 21:02:31Z) — 10 minute(s) behind, 0 report(s) still undelivered when the pass began
2026-10-21 17:58:20Z - Loki query_range [2026-10-21 17:52:20Z, 2026-10-21 17:58:20Z) — 6 minute(s) behind, 0 report(s) still undelivered when the pass began

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver.Plugins/issues/2156, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/log-pipeline-behind-memex. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site df7022d82cca1b39: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
6h
PR unite (30g)
967

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.