Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Durable seed read for a notification hub's MeshDataSource never returned; DataContext initialization hit the 115s bound and retired the activation

Đang mở
  • Có gì không ổn?
#7,082 16 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
25/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
csharp
Lĩnh vực
backend

Hướng nghiên cứu

Read src/MeshWeaver.Graph/MeshNodeTypeSource.cs and the relevant initialization and persistence code under src/MeshWeaver.Data; then review Doc/Architecture/InitializationGates for the initialization contract and IIoPool rule. Compare another pod’s logs for the same window, DB query timing, and this pod’s GC pause and IIoPool metrics to distinguish the candidate causes. Done means establishing the cause and whether the seed reads completed or were abandoned.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug sev:M

Observed

On pod memex-portal-deployment-8556887fd9-hwx6z, the DataContext initialization of hub Admin/_Notification/BVwQ4OUFukurFkS6PMH9Yw (an admin Notification node, "Type 'Store/Install' is serving a fallback page") did not complete within the 115s initialization bound. The hub's MeshDataSource had one MeshNode stream (stream B4-PZRYrqUSlaKeDITXZYg, owner ds/Admin/_Notification/BVwQ4OUFukurFkS6PMH9Yw, partition none, sync stream sync/7jy9B7NSbEm0iP3qj-tM4g) that never produced a first frame, because the durable seed read of the hub's own path had been outstanding for ~117s and had not come back. The activation was retired rather than latched — the log line itself states the next access re-creates the hub and nothing retries on its own.

Where and when

This fingerprint fired once, but it is one instance of a burst on the same pod only at the same log site: many sibling Admin/_Notification hubs show the identical shape (1 of 2 data sources stuck on a durable seed read outstanding ~105–124s) between ~08:02:08 and ~08:02:54 (e.g. fingerprint 3cc8bde7680a1a0e with 19 occurrences, 8414cc491dd7a00d, and others). The MessageHub log site recorded sync-hub BuildupAction timeouts on the same pod in the same window (incident 016a5164db586ae3, filed as Systemorph/MeshWeaver#7081), on the same sync/ stream ids (e.g. sync/v2ragjHgc0W_rO3iJVT8MA appears in both). Back-dating the outstanding-read durations puts the stalled reads' start at ~08:00.

Immediately before that, at 07:59:04–08:00:13, the same pod's process_progress health check reported GC-bound with pause share rising 53% → 81% → 85% (incident afc6241b330977b0, which references MeshWeaver#2194). The stalled seed reads were therefore issued into the tail of a period in which the process was reporting it was not making progress.

Impact

Admin-facing notification hubs on one pod only. The DataContext timeout and activation retirement behaved as designed (liveness bound made the failure observable; the next access re-creates the hub), so the user-visible damage is a delayed notification hub, not lost data. No other pod shows this burst; no evidence of tenant-facing paths being affected.

Not established

  • Cause. The evidence does not distinguish a pod-local stall from a storage-side stall; see candidates below.
  • Whether the outstanding seed reads ever completed after the activation was retired, or were abandoned with it — no completion log line was observed.
  • Whether it still occurs — single occurrence in this fingerprint at triage time.
  • Whether other log sites show the same stall shape in this window beyond the DataContext and MessageHub sites already cited.

Candidate causes and the reading that separates them

  1. GC-bound pod starved the storage-read path. The pod reported GC pause shares up to 85% minutes before the burst; reads issued ~08:00 never returned within the 115s bound. Predicts: the burst is bounded to this pod and to the GC episode's window. Discriminating read: the pod's GC pause-share timeline 08:00–08:05 — if the burst stops when the GC episode ends, this cause is confirmed.
  2. Storage-backend stall. The durable seed read goes through IMeshNodePersistenceCore.Read (the async DB leaf is pooled through IIoPool inside the persistence core). Predicts: other pods reading the same storage at the same instants show the same stalled reads, and the database shows slow or abandoned queries around 08:00–08:02. Discriminating read: another pod's logs for the same window, or DB-side query timing.
  3. A defect in the seed-read path itself (a read that never completes, holding its pool slot). Evidence against: dozens of independent hubs stalled simultaneously, inconsistent with a per-hub deadlock but consistent with pool saturation. Discriminating read: IIoPool depth/saturation metrics on this pod around 08:00.

Related, not duplicates

  • MeshWeaver#7081 (sync hub BuildupAction hang, same pod and window) — different log site, plausibly the same underlying pod-wide stall.
  • MeshWeaver#2194 (GC-bound process progress) — the health-check incident covering the GC episode this burst rides on; if this ticket's cause resolves to #2194, it can be closed as a downstream symptom.

Where to look

  • src/MeshWeaver.Graph/MeshNodeTypeSource.cs — the durable seed leg of Initialize (IMeshNodePersistenceCore.Read).
  • src/MeshWeaver.Data — the persistence core's seed read and the MeshWeaver.Data.DataContext initialization bound that produced this log line.
  • Doc/Architecture/InitializationGates (in-repo, Doc/Architecture) — the MeshNodeTypeSource init contract and the IIoPool rule for async DB leaves.
Evidence
Fingerprint 45c732efff5849cb
Category MeshWeaver.Data.DataContext
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-8556887fd9-hwx6z
Occurrences 1
First seen 2026-10-23 08:02:13Z
Last seen 2026-10-23 08:02:13Z
Routing not determined — no configured route matches the category MeshWeaver.Data.DataContext. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-23 08:02:13Z memex-portal-deployment-8556887fd9-hwx6z fail: MeshWeaver.Data.DataContext[0]
      DataContext initialization TIMED OUT for Admin/_Notification/BVwQ4OUFukurFkS6PMH9Yw. Retiring this activation instead of latching it FAILED — the next access re-creates it; nothing retries on its own.
      System.TimeoutException: Hub 'Admin/_Notification/BVwQ4OUFukurFkS6PMH9Yw' DataContext initialization did not complete within 115s. Still waiting on 1 of 2 data source(s): 'Admin/_Notification/BVwQ4OUFukurFkS6PMH9Yw (MeshDataSource)' — 1 of 1 stream(s) never produced a first frame (sync/7jy9B7NSbEm0iP3qj-tM4g stream=B4-PZRYrqUSlaKeDITXZYg owner=ds/Admin/_Notification/BVwQ4OUFukurFkS6PMH9Yw partition=(none)), type-source legs still outstanding: B4-PZRYrqUSlaKeDITXZYg/MeshNode [durable seed read of 'Admin/_Notification/BVwQ4OUFukurFkS6PMH9Yw' outstanding for 116.9s — a storage read that has not come back]. Settled: _Comment (MeshDataSource).

Opened automatically from Admin/_LogIncident/45c732efff5849cb. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 36a14eef87f2ff9c: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Ngôn ngữ chính
C#
Star
12
Fork
5
Merge trung bình
6 giờ 39 phút
Pull request đã merge (30 ngày)
891

Chuẩn bị môi trường

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Systemorph/MeshWeaver

Tất cả issue của Systemorph/MeshWeaver

Issue tương tự

Thêm issue về C#

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.