Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

mw-log-watcher stopped polling memex for ~9 minutes: one pass read a window far wider than the poll interval with an empty delivery queue

Open
  • Is something wrong?
#7,043 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp, grafana, kubernetes
Domain
backend, observability

Research direction

Start with LogWatchWorker and WatcherState.CursorFor in src/MeshWeaver.LogWatcher, then read the "Delivery guarantees" and "The watcher tickets its own blind spots" sections of LogWatchTriage. Check the watcher pod logs with the provided kubectl command across a lag window to distinguish a restart from a stalled poll loop. Done means the cause is established and the watcher avoids repeated polling gaps; verify behavior against the cursor and MaxCatchUp contract.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

The mw-log-watcher serving namespace memex went roughly nine poll intervals without completing a pass: its next Loki query_range covered a window far wider than the 1-minute PollInterval, and the store answered it — so nothing was lost, but nothing in that stretch was read, grouped, queued or delivered until the late pass ran, and every red log in it was ticketed that late. This is the watcher ticketing its own blind spot (the Pipeline behind-finding), and the delay directly corrupts the incident record: an incident's createdDate is its delivery time, so a lag this size makes the record read as "ticketing stopped".

Probable cause

The message offers two candidates — the watcher wasn't polling, or a delivery backlog held the pass. The sample rules the second one out: zero reports were still undelivered when the pass began, so the delivery queue was empty and delivery cannot have blocked the poll loop. That leaves the first: the watcher was not polling for that stretch. Most likely a pod restart or a stalled/blocked poll loop (a hung Loki query, scheduling starvation, or a crash-loop window on the monitoring Deployment). Confidence: moderate — a single occurrence with one sample and no pod list on the incident; the empty-queue reading is solid, but distinguishing restart from stall needs the watcher's own pod log (kubectl -n monitoring logs deploy/mw-log-watcher), which is not readable from the portal.

Impact

Single occurrence, single namespace, no data lost — the store answered the wide window. This is a delayed-detection defect, not an outage. But it is the precursor of the condition that does lose data: a lag that keeps growing ends at the MaxCatchUp floor, where the skipped window is never read. That exact endpoint already happened on the sibling namespace — see Admin/_LogIncident/log-window-skipped-memex-cloud (filed as MeshWeaver.Plugins#2156, superseded by MeshWeaver#4326). Worth watching whether this recurs with a growing window rather than a one-off restart blip.

Where to look

  1. The mw-log-watcher Deployment (namespace monitoring) serving memex — its pod log across the lag window, to tell restart-crash from loop-stall.
  2. The poll loop and cursor advance in src/MeshWeaver.LogWatcher (MeshWeaver.Plugins): LogWatchWorker, WatcherState.CursorFor — check whether anything between "advance cursor" and the next poll can block (a slow query_range with no timeout, or a synchronous step inside the loop).
  3. LogWatchTriage — sections "Delivery guarantees" and "The watcher tickets its own blind spots" describe the cursor/MaxCatchUp contract this finding guards.

Already known?

Not a duplicate, but the same log site as two filed siblings: log-burst-header-only-memex and log-window-skipped-memex-cloud (site fold df7022d82cca1b39, MeshWeaver.LogWatcher.Pipeline). Those are different findings (bodyless captures; a MaxCatchUp-floored skip); this one is the behind-but-recoverable predecessor of the latter.

Evidence
Fingerprint log-pipeline-behind-memex
Category MeshWeaver.LogWatcher.Pipeline
Severity Error
Namespace memex
Occurrences 4
First seen 2026-10-05 10:29:44Z
Last seen 2026-10-21 17:58:20Z
Routing not determined — no configured route matches the category MeshWeaver.LogWatcher.Pipeline. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 10:38:33Z - Loki query_range [2026-10-05 10:29:44Z, 2026-10-05 10:38:33Z) — 9 minute(s) behind, 0 report(s) still undelivered when the pass began
2026-10-05 10:41:48Z - Loki query_range [2026-10-05 10:35:21Z, 2026-10-05 10:41:48Z) — 6 minute(s) behind, 89 report(s) still undelivered when the pass began
2026-10-05 21:02:31Z - Loki query_range [2026-10-05 20:52:51Z, 2026-10-05 21:02:31Z) — 10 minute(s) behind, 0 report(s) still undelivered when the pass began
2026-10-21 17:58:20Z - Loki query_range [2026-10-21 17:52:20Z, 2026-10-21 17:58:20Z) — 6 minute(s) behind, 0 report(s) still undelivered when the pass began

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver.Plugins/issues/2156, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/log-pipeline-behind-memex. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site df7022d82cca1b39: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
6h 29m
Merged PRs (30d)
914

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.