A shard whose drive is full cannot free space

Aperta
#4,319 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug I4 neofs-storage S4 U4

Expected Behavior

Deleting data frees space, on a full drive most of all.

Current Behavior

GC removes objects from the metabase first and stops if that fails
(shard/delete.go#L43-L46:
return err // stop on metabase error ?). The metabase is on the same
drive, so on a full one the write fails and nothing is deleted, on every
GC pass:

warn	shard/gc.go:172	can't delete objects	{"shard_id": "Ck99uKR2FLWR7oYAH3Vfyz", "error": "write /drives/d0/meta: no space left on device"}

0.58.2, four storage nodes with two shards each, every shard on its own
1 GiB ext4 filesystem (metabase and payload together), REP 2. 4 MiB
objects were written until every PUT failed; four of the eight
filesystems were then 100% full. Then the container was removed
(neofs-cli container delete). Nine minutes later:

Filesystem Used Objects deleted by GC
s01 d0, s03 d0, s03 d1, s04 d0 (not full before) 1% about 237 each
s01 d1, s02 d0, s02 d1, s04 d1 (full before) 100% 0, the warning above on every pass

Node s02, with both shards full, can store nothing and free nothing, and
stays that way.

Possible Solution

Keep a reserve: stop accepting writes on a shard before its filesystem is
full, so that the metabase can still be written. Or free payload first
when the metabase write fails for lack of space.

Steps to Reproduce (for bugs)

  1. Storage nodes whose shards are on small filesystems of their own.
  2. PUT objects until all PUTs fail with no space left on device.
  3. Delete the container (or the objects) and watch the node's log and df.

Regression

Not known.

Your Environment

  • Version used: 0.58.2 (first seen on 0.58.1)
  • Server setup and configuration: 4 storage nodes, 2 shards each (fstree and metabase on one ext4 filesystem of 1 GiB per shard, write-cache disabled), 1 inner ring node, in Docker on one machine
  • Operating System and version (uname -a): Linux 6.8.0 aarch64 (Ubuntu 24.04)
Lingua principale
Go
Stelle
37
Fork
56
Merge medio
1g 21h
PR unite (30g)
46

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di nspcc-dev/neofs-node

Tutte le issue di nspcc-dev/neofs-node

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.