Activation-record OPEN for MeshWeaver.AI.Codex fails with errno 11 (EAGAIN) on one memex-cloud pod — first burst carrying the #7689 {path} diagnostic; module dropped from that read

Đã đóng
  • Có gì không ổn?
#7,797 3 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@rbuergi đang làm issue này rồi.

Từ ngày 3/11/2026.

  • #7873 của @rbuergi — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
25/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Đình trệ
Công nghệ
csharp, linux
Lĩnh vực
backend, observability

Hướng nghiên cứu

Start with MeshWeaver.PluginCatalog.ModuleLandingService and ModuleActivationSidecar.Read / DeriveEntry in ModuleActivation.cs, then review the #7689 and #7690 findings. Compare the affected pod's node and image with co-located pods, and inspect SMB client counters and kernel logs around 2026-11-01 15:08:42Z. Done means identifying whether the OPEN failure came from a node-local lock or SMB transport and documenting the evidence; linked PR #7873 is already open.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug sev:L

Observed

One fail line from MeshWeaver.PluginCatalog.ModuleLandingService[0] on pod memex-portal-deployment-6d66679bcb-2748t (memex-cloud): opening a module activation record under {path}/ failed with IOException, HResult 11 (0x0000000B) at step OPEN (open, stat and the advisory shared lock) — module MeshWeaver.AI.Codex cannot be read whole.

This is the first burst of this log site that carries the {path} diagnostic built after #7689, and the errno is itself evidence. Per the documented semantics (Module Activation Head Ownership, "What 'being used by another process' means here (#7689)"): errno 11 (EAGAIN) at OPEN is either a node-local exclusive flock (the share is mounted nobrl, so a flock never leaves the node) or a transient SMB status mapped to EAGAIN (STATUS_INSUFFICIENT_RESOURCES, STATUS_IO_TIMEOUT, STATUS_RETRY). A genuine server-side sharing violation maps to EBUSY and reads differently; 11 at READ can only be the transport. The current image takes no exclusive lock on any activation record, and readers open with FileShare.ReadWrite | FileShare.Delete.

The designed consequence fired two seconds later on the same pod, on the companion fingerprint de924b4549cd1052 (issue #7690): MeshWeaver.AI.Codex was dropped whole from that read — the fail-closed rule; the next read that sees the records whole restores it.

Where and when

A single occurrence, on a single pod, for a single module — against the 2026-10-30 burst of this site (#7689 / #7690: two pods, two modules, one instant). The pod carries a third distinct ReplicaSet hash (6d66679bcb; the October burst saw 8597b8c8f4 and d6c9dd984), consistent with the deployment's roll history continuing to produce mixed-image windows.

Impact

One degraded activation read on one pod: MeshWeaver.AI.Codex absent from that read, and ProposeModuleSet refuses to propose from a faulted read. Judged low on the numbers — a single occurrence, handled by the documented fail-closed rule with documented self-healing. Not established: whether this read was a boot read, where the drop would hold for the process's lifetime; if it was, re-rule upward.

Not established

  • The cause: errno 11 at OPEN is ambiguous between a node-local lock and the transport (below).
  • Whether the record reads whole now — no further line on this fingerprint has arrived, which is weak evidence of self-healing, not proof.
  • Whether a landing of MeshWeaver.AI.Codex was in flight at that instant (the incident carries no delivery or publish record for the window).
  • Which process, if any, held the file; whether other modules' record reads faulted in the same window.

Candidate causes and the reading that separates them

  1. Node-local exclusive flock by a process on the reader's own node. The nobrl mount keeps the flock in the node's kernel, so the holder shares a node with memex-portal-deployment-6d66679bcb-2748t. Since the current image takes no exclusive lock on any activation record, a holder points to an image predating that (a mixed-image window — the residual Publishing A File On A Shared Volume names) or a non-portal process on the node. Predicts: bursts clustered on landing instants by co-located pods, stopping once every replica runs the newer image. Separating reading: which pods shared that node at the instant and their image versions; whether a MeshWeaver.AI.Codex landing was publishing then.
  2. SMB transport throttling or timeout. The client's mapping of STATUS_INSUFFICIENT_RESOURCES / STATUS_IO_TIMEOUT / STATUS_RETRY to -EAGAIN. Predicts: faults uncorrelated with landings, across pods and modules. Separating reading: the node's SMB client session counters / kernel log around the instant; whether reads of OTHER modules' records faulted in the same window.

What this burst narrows: an off-node holder with an exclusive share mode reads EBUSY at the server, so whatever refused this OPEN acted on the reader's own node or on the transport — the #2190 copy-fallback shape by a remote pod is not what this errno shows (a co-located pod's copy fallback is not excluded by it).

Where to look

  • MeshWeaver.PluginCatalog.ModuleLandingService — the record-read log site; ModuleActivationSidecar.Read / DeriveEntry in ModuleActivation.cs — the enumeration of activation.d/<Name>/ and the fail-closed drop.
  • Module Activation Head Ownership — the #7689 section defining the errno semantics this line now carries.
  • The prior tickets of the same site and fault class: #7689 (the read-fault lines, 2026-10-30 burst, before the {path} diagnostic existed) and #7690 (the module-drop lines — including this burst's follow-on for MeshWeaver.AI.Codex on the same pod). This incident is that fault class's next burst, now with the {path} detail those tickets lacked; a human may fold it into #7690 quickly.
Evidence
Fingerprint 22de0814b8838f70
Category MeshWeaver.PluginCatalog.ModuleLandingService
Severity Error
Exception IOException
Namespace memex-cloud 1 occurrence(s), 2026-11-01 15:08:42Z → 2026-11-01 15:08:42Z; pods memex-portal-deployment-6d66679bcb-2748t
Occurrences 1
First seen 2026-11-01 15:08:42Z
Last seen 2026-11-01 15:08:42Z
Routing not determined — no configured route matches the category MeshWeaver.PluginCatalog.ModuleLandingService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-11-01 15:08:42Z memex-portal-deployment-6d66679bcb-2748t fail: MeshWeaver.PluginCatalog.ModuleLandingService[0]
      Module record '{path}' could not be read (IOException: The process cannot access the file '{path}' because it is being used by another process. [step: OPEN (open, stat and the advisory shared lock); path: '{path}'; error number (HResult): 11 (0x0000000B). On Linux, 11 at OPEN is either a node-local exclusive flock or a transient SMB status mapped to EAGAIN; 11 at READ can only be the SMB transport (MeshWeaver#7689).]) — module 'MeshWeaver.AI.Codex' cannot be read whole.

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/7689, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/22de0814b8838f70. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 341962cffb3768ba: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Ngôn ngữ chính
C#
Star
12
Fork
5
Merge trung bình
5 giờ 14 phút
Pull request đã merge (30 ngày)
957

Chuẩn bị môi trường

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Systemorph/MeshWeaver

Tất cả issue của Systemorph/MeshWeaver

Issue tương tự

Thêm issue về C#

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.