Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

FileSystemAssemblyStore refuses NodeType compile publication on ENOSPC — /data/assembly-cache out of space on memex-portal pod

Closed
  • Is something wrong?
#6,940 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #6969 by @rbuergi — merged

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp, kubernetes, linux
Domain
infrastructure, observability-sre

Research direction

Start with src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs, especially OpenTempForWrite, then inspect cache files and pod metrics. Run df and inode checks for /data/assembly-cache, and review /data/assembly-cache/Store_* retention. Done means the evidence distinguishes capacity exhaustion, inode exhaustion, stale cache accumulation, or shared-node pressure, with the affected assembly publication path verified.

Written by the indexing model from the issue text.

Description

bug sev:M

Observed

MeshNodeCompilationService compiled three Store NodeTypes and FileSystemAssemblyStore refused publication, marking each compile FAILED and minting no Release, because writing the compiled bytes failed with System.IO.IOException: No space left on device while opening the temp PDB file under /data/assembly-cache. Affected NodeTypes: Store/CouponClaim@v73, Store/CouponClaimSlot@v22, Store/CouponJourneyHandler@v75 — all on pod memex-portal-deployment-567c764557-mpx9l.

The failing write path, read from the stack traces: FileSystemAssemblyStore.PutWithLocationCore → AtomicFileWrite.PublishBytesWith → AtomicFileWrite.WriteDurably → AtomicFileWrite.OpenTempForWrite, throwing at Interop.ThrowExceptionForIoErrno on FileStream construction — the ENOSPC surfaced at temp-file open, not at write or flush.

Where and when

Single pod (memex-cloud namespace), 3 occurrences inside a 4 ms window at 05:46:16Z on 2026-10-18 — three near-simultaneous NodeType compilations against a volume that was already out of space at that instant. No earlier or later occurrences exist in this fingerprint, and no prior disk-space incident is recorded in the incident queue. All three NodeTypes are Store/* coupon node types at high version numbers (v73/v22/v75), consistent with a recompile wave over the Store module; whether version churn contributed to cache growth is a candidate, not established.

Impact

The three named NodeType versions have no published assembly and no Release on that pod. The failure is loud (publication refused, compile marked FAILED), not silent. Scale so far: one pod, one instant, three node types.

Not established

  • Whether the exhaustion is byte-based or inode-based — Linux ENOSPC covers both.
  • Whether the condition persisted after the burst or cleared (e.g. by eviction or restart); no later samples exist.
  • Whether other pods were affected; only this pod appears.
  • Whether the full disk also failed other write paths on the pod (only this log site was fingerprinted).
  • What the store serves in place of the refused versions and whether portal behaviour degrades beyond the missing Releases — not read.

Candidate causes and the reading that separates them

  1. Capacity exhaustion of the cache volume — predicts df on /data/assembly-cache on that pod showing ~0 bytes available at 05:46Z. Reading: the pod's ephemeral-storage usage metrics or container-status events around that timestamp.
  2. Inode exhaustion — same errno — predicts inode usage at 100% while bytes remain. Reading: inode-level usage on that filesystem.
  3. Accumulation of superseded versions in the cache (the version numbers in the temp paths imply a long per-node-type history) — predicts the /data/assembly-cache/Store_* directories hold many retained versions and dominate usage. Reading: per-directory file counts and sizes in the cache, and whether the store prunes superseded versions at all.
  4. External tenant on the node or a shared volume — predicts ENOSPC in other pods' logs at the same instant or node-level disk-pressure events. Reading: the node's ephemeral-storage pressure events for that window.

Where to look

Start from the frames read in the samples: MeshWeaver.Mesh.Persistence.AtomicFileWrite (OpenTempForWrite at line 167, WriteDurably at line 195, PublishBytesWith at line 135, in src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs) and MeshWeaver.Compiler.FileSystemAssemblyStore.PutWithLocationCore. The MeshWeaver framework source was not found in this mesh — the paths above come verbatim from the stack traces.

Evidence
Fingerprint fc3f9e106575dad3
Category MeshWeaver.Graph.Configuration.MeshNodeCompilationService
Severity Error
Exception System.IO.IOException
Top frame Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
Namespace memex-cloud
Pods memex-portal-deployment-567c764557-mpx9l
Occurrences 3
First seen 2026-10-18 05:46:16Z
Last seen 2026-10-18 05:46:16Z
Routing not determined — no configured route matches the category MeshWeaver.Graph.Configuration.MeshNodeCompilationService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-18 05:46:16Z memex-portal-deployment-567c764557-mpx9l fail: MeshWeaver.Graph.Configuration.MeshNodeCompilationService[0]
      AssemblyStore REFUSED the publication for Store/CouponClaim@v73: the compiled bytes did not land, so the compile is FAILED and no Release is minted
      System.IO.IOException: No space left on device : '/data/assembly-cache/Store_CouponClaim/.tmp-cf7861814dcc453dbc903146bc8a6592-v73-c003e001-e505ce7a0274.pdb.tmp'
         at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
         at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
         at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
         at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.OpenTempForWrite(String tempPath) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 167
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.WriteDurably(String tempPath, Byte[] bytes, Func`2 openTemp) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 195
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.PublishBytesWith(String filePath, Byte[] bytes, Func`2 openTemp) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 135
         at MeshWeaver.Compiler.FileSystemAssemblyStore.PutWithLocationCore(String nodeTypePath, Int64 version, Byte[] assemblyBytes, Byte[] pdbBytes) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshW…[truncated]
2026-10-18 05:46:16Z memex-portal-deployment-567c764557-mpx9l fail: MeshWeaver.Graph.Configuration.MeshNodeCompilationService[0]
      AssemblyStore REFUSED the publication for Store/CouponClaimSlot@v22: the compiled bytes did not land, so the compile is FAILED and no Release is minted
      System.IO.IOException: No space left on device : '/data/assembly-cache/Store_CouponClaimSlot/.tmp-0666d1d2c0d9459cbce069b71be1133d-v22-c003e001-7fef08509a2b.pdb.tmp'
         at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
         at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
         at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
         at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.OpenTempForWrite(String tempPath) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 167
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.WriteDurably(String tempPath, Byte[] bytes, Func`2 openTemp) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 195
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.PublishBytesWith(String filePath, Byte[] bytes, Func`2 openTemp) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 135
         at MeshWeaver.Compiler.FileSystemAssemblyStore.PutWithLocationCore(String nodeTypePath, Int64 version, Byte[] assemblyBytes, Byte[] pdbBytes) in /home/runner/work/MeshWeaver/MeshWeaver/s…[truncated]
2026-10-18 05:46:16Z memex-portal-deployment-567c764557-mpx9l fail: MeshWeaver.Graph.Configuration.MeshNodeCompilationService[0]
      AssemblyStore REFUSED the publication for Store/CouponJourneyHandler@v75: the compiled bytes did not land, so the compile is FAILED and no Release is minted
      System.IO.IOException: No space left on device : '/data/assembly-cache/Store_CouponJourneyHandler/.tmp-151ee760e81a46138de89a0069bfbc69-v75-c003e001-f838e43ca260.pdb.tmp'
         at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
         at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
         at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
         at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.OpenTempForWrite(String tempPath) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 167
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.WriteDurably(String tempPath, Byte[] bytes, Func`2 openTemp) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 195
         at MeshWeaver.Mesh.Persistence.AtomicFileWrite.PublishBytesWith(String filePath, Byte[] bytes, Func`2 openTemp) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Mesh.Contract/Persistence/AtomicFileWrite.cs:line 135
         at MeshWeaver.Compiler.FileSystemAssemblyStore.PutWithLocationCore(String nodeTypePath, Int64 version, Byte[] assemblyBytes, Byte[] pdbBytes) in /home/runner/work/MeshWeaver/Me…[truncated]

Opened automatically from Admin/_LogIncident/fc3f9e106575dad3. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 806a78e8b3b2a17b: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 55m
Merged PRs (30d)
968

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.