Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

memex-cloud: shared /data volume full for 33 h after #6940 closed; still filling, and no alarm or incident reports it

Aperta
  • Qualcosa non va?
#6,983 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp, kubernetes
Ambito
infrastructure, observability

Direzione di ricerca

Start by reading the referenced fleet watch and storage_capacity / data_volume_free_space status implementation, then trace where it creates Ops/Issue records. Check the retention and eviction logging path and the cluster’s Loki log-level configuration. Done means degraded storage creates an issue record and retention or eviction outcomes are visible in Loki; the volume investigation and recovery steps also need operational coordination.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:hosting area:storage bug sev:H

Why MeshWeaver, and prior art

The assembly store, module landing, retention and the fleet watch are MeshWeaver core code, so this issue belongs in MeshWeaver core. Prior art checked: #6940 (closed by #6969 on 2026-10-18, root cause not established), #6969, #3741, #2086, Memex #253 and #562. None is open, and none covers the condition after #6969.

Observed

  1. On memex-cloud, /data, /data/assembly-cache and /data/prebuilt-bundles (the shared 128 GiB memex-data volume) have 2–3 MiB free of 131,072 MiB on all 7 replicas (Ops/Status/memex-cloud, 2026-10-19 16:41 CEST). At 13:36 CEST, 10 MiB was free. The free space is still falling.
  2. The volume has been full since at least 2026-10-18 07:25 CEST (first out-of-space line, /data/modules/sets/…proposed.json, from the #6969 timeline).
  3. The running image memex-portal-ai:3.1.11217 (c6c0ad1d) contains #6969. The replicas booted at 2026-10-19 03:36 CEST, and the volume is still full on all of them. The same-version builds that #6969 limits were therefore not the main cause, or eviction does not run on this volume.
  4. memex (memex.systemorph.com) runs the same image and does not show this condition.

Gap in alerting

  1. The triage bot creates incidents from Error log lines only. No out-of-space line was logged after 2026-10-18 08:42 CEST, so incident fc3f9e106575dad3 was not reopened and #6940 stays closed.
  2. storage_capacity and data_volume_free_space report Degraded in Ops/Status, but the fleet watch creates no Ops/Issue record for them.
  3. Retention and eviction reports log at Information level, and the cluster sends only Warning and above to Loki (#6969). What deletes data from the volume, and what writes to it, are therefore not visible.

Impact

  1. All 7 replicas share the volume, so every replica is affected at once.
  2. NodeType compiles and module landings write into the volume, and a write on a full volume is refused. 17 modules are landed but not loaded, including MeshWeaver.Blazor.Chat and three MeshWeaver.AI.* modules.
  3. On 2026-10-18, a pod that had just rebooted could not write its claim file under /data/assembly-cache/.generations/. The next roll or restart can fail in the same way. This is not tested.
  4. A possible link to the chat error #6980 (Cannot convert type ThreadViewModel) on the same replicas is not established.

Not established

  1. Which subdirectory holds the space: modules/, assembly-cache/, prebuilt-bundles/ or the NuGet cache.
  2. Which process still writes to the volume.
  3. Whether retention and eviction run on memex-cloud and what they remove.
  4. Why memex does not fill on the same image. Its 3 replicas compared with memex-cloud's 7 is one candidate.

Requested

  1. Read per-directory usage on the memex-data volume through a governed, read-only instance action.
  2. Free the volume, then recompile every type that reports a refusal.
  3. Make the fleet watch create an Ops/Issue record when storage_capacity or data_volume_free_space is Degraded.
  4. Log retention and eviction outcomes at a level that reaches Loki.
Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 57m
PR unite (30g)
967

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.