Activation-record OPEN for MeshWeaver.AI.Codex fails with errno 11 (EAGAIN) on one memex-cloud pod — first burst carrying the #7689 {path} diagnostic; module dropped from that read

Start with MeshWeaver.PluginCatalog.ModuleLandingService and ModuleActivationSidecar.Read / DeriveEntry in ModuleActivation.cs, then review the #7689 and #7690 findings. Compare the affected pod's node and image with co-located pods, and inspect SMB client counters and kernel logs around 2026-11-01 15:08:42Z. Done means identifying whether the OPEN failure came from a node-local lock or SMB transport and documenting the evidence; linked PR #7873 is already open.

Observed

One fail line from MeshWeaver.PluginCatalog.ModuleLandingService[0] on pod memex-portal-deployment-6d66679bcb-2748t (memex-cloud): opening a module activation record under {path}/ failed with IOException, HResult 11 (0x0000000B) at step OPEN (open, stat and the advisory shared lock) — module MeshWeaver.AI.Codex cannot be read whole.

This is the first burst of this log site that carries the {path} diagnostic built after #7689, and the errno is itself evidence. Per the documented semantics (Module Activation Head Ownership, "What 'being used by another process' means here (#7689)"): errno 11 (EAGAIN) at OPEN is either a node-local exclusive flock (the share is mounted nobrl, so a flock never leaves the node) or a transient SMB status mapped to EAGAIN (STATUS_INSUFFICIENT_RESOURCES, STATUS_IO_TIMEOUT, STATUS_RETRY). A genuine server-side sharing violation maps to EBUSY and reads differently; 11 at READ can only be the transport. The current image takes no exclusive lock on any activation record, and readers open with FileShare.ReadWrite | FileShare.Delete.

The designed consequence fired two seconds later on the same pod, on the companion fingerprint de924b4549cd1052 (issue #7690): MeshWeaver.AI.Codex was dropped whole from that read — the fail-closed rule; the next read that sees the records whole restores it.

Where and when

A single occurrence, on a single pod, for a single module — against the 2026-10-30 burst of this site (#7689 / #7690: two pods, two modules, one instant). The pod carries a third distinct ReplicaSet hash (6d66679bcb; the October burst saw 8597b8c8f4 and d6c9dd984), consistent with the deployment's roll history continuing to produce mixed-image windows.

Impact

One degraded activation read on one pod: MeshWeaver.AI.Codex absent from that read, and ProposeModuleSet refuses to propose from a faulted read. Judged low on the numbers — a single occurrence, handled by the documented fail-closed rule with documented self-healing. Not established: whether this read was a boot read, where the drop would hold for the process's lifetime; if it was, re-rule upward.

Not established

• The cause: errno 11 at OPEN is ambiguous between a node-local lock and the transport (below).
• Whether the record reads whole now — no further line on this fingerprint has arrived, which is weak evidence of self-healing, not proof.
• Whether a landing of MeshWeaver.AI.Codex was in flight at that instant (the incident carries no delivery or publish record for the window).
• Which process, if any, held the file; whether other modules' record reads faulted in the same window.

Candidate causes and the reading that separates them

1. Node-local exclusive flock by a process on the reader's own node. The nobrl mount keeps the flock in the node's kernel, so the holder shares a node with memex-portal-deployment-6d66679bcb-2748t. Since the current image takes no exclusive lock on any activation record, a holder points to an image predating that (a mixed-image window — the residual Publishing A File On A Shared Volume names) or a non-portal process on the node. Predicts: bursts clustered on landing instants by co-located pods, stopping once every replica runs the newer image. Separating reading: which pods shared that node at the instant and their image versions; whether a MeshWeaver.AI.Codex landing was publishing then.
2. SMB transport throttling or timeout. The client's mapping of STATUS_INSUFFICIENT_RESOURCES / STATUS_IO_TIMEOUT / STATUS_RETRY to -EAGAIN. Predicts: faults uncorrelated with landings, across pods and modules. Separating reading: the node's SMB client session counters / kernel log around the instant; whether reads of OTHER modules' records faulted in the same window.

What this burst narrows: an off-node holder with an exclusive share mode reads EBUSY at the server, so whatever refused this OPEN acted on the reader's own node or on the transport — the #2190 copy-fallback shape by a remote pod is not what this errno shows (a co-located pod's copy fallback is not excluded by it).

Where to look

• MeshWeaver.PluginCatalog.ModuleLandingService — the record-read log site; ModuleActivationSidecar.Read / DeriveEntry in ModuleActivation.cs — the enumeration of activation.d/<Name>/ and the fail-closed drop.
• Module Activation Head Ownership — the #7689 section defining the errno semantics this line now carries.
• The prior tickets of the same site and fault class: #7689 (the read-fault lines, 2026-10-30 burst, before the {path} diagnostic existed) and #7690 (the module-drop lines — including this burst's follow-on for MeshWeaver.AI.Codex on the same pod). This incident is that fault class's next burst, now with the {path} detail those tickets lacked; a human may fold it into #7690 quickly.

Evidence

2026-11-01 15:08:42Z memex-portal-deployment-6d66679bcb-2748t fail: MeshWeaver.PluginCatalog.ModuleLandingService[0]
      Module record '{path}' could not be read (IOException: The process cannot access the file '{path}' because it is being used by another process. [step: OPEN (open, stat and the advisory shared lock); path: '{path}'; error number (HResult): 11 (0x0000000B). On Linux, 11 at OPEN is either a node-local exclusive flock or a transient SMB status mapped to EAGAIN; 11 at READ can only be the SMB transport (MeshWeaver#7689).]) — module 'MeshWeaver.AI.Codex' cannot be read whole.

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/7689, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

_{Opened automatically from Admin/_LogIncident/22de0814b8838f70. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 341962cffb3768ba: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}