Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

NBD device-release race + partscan noise: isDeviceFree predicate too weak, causes dmesg I/O errors

Aperta
  • Qualcosa non va?
#3,652 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
65/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, linux
Ambito
infrastructure, operating-systems

Direzione di ricerca

Start in pool.go at DevicePool.isDeviceFree and inspect the existing Go tests covering device release and ReclaimLeaked. Verify the current pid and size checks, then cover the inflight and holders signals with fallback behavior for missing sysfs files. Done means a device is reused only after the required quiescence checks pass, while older kernels do not wedge the pool.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Symptom

On nodes with rapid sandbox create/destroy churn (dev-sandbox-2, kernel 6.14.0-37), dmesg fills with NBD I/O errors clustered exactly at the moment one sandbox is being torn down while its NBD device is still releasing and a new sandbox's allocation request arrives:

block nbd76: Receive control failed (result -32)
nbd78: detected capacity change from 0 to 47847424
block nbd76: NBD_DISCONNECT
block nbd76: Send disconnect failed -32
I/O error, dev nbd76, sector 0 op 0x0:(READ) ...
Buffer I/O error on dev nbd76, logical block 0, async page read
ldm_validate_partition_table(): Disk read failed.
nbd76: unable to read partition table

Root cause

1. isDeviceFree predicate is too weak (pool.go)

DevicePool.isDeviceFree decides a slot is free from only two signals:

  1. /sys/block/nbdX/pid absent (no connected server), and
  2. /sys/block/nbdX/size == 0.

But NBD disconnect completes in two stages:

  • Synchronous stage: Status.Connected flips false, the pid file disappears, and nbd_bdev_reset calls set_capacity(disk, 0) so size reads 0. These finish inside the disconnect call path.
  • Asynchronous stage: completion/cancellation of in-flight blk_mq requests, page-cache invalidation (__invalidate_device), reclaim of readahead / partition-scan reads, and exit of the recv work on nbd->recv_workq. These run on a workqueue and are not guaranteed to be done at the instant size hits 0.

So size==0 && no pid only proves the capacity was reset — it does not prove the device is quiescent. The dmesg Receive control failed (result -32 / -EPIPE) and I/O error sector 0 lines are exactly the async-stage probe reads / in-flight requests hitting a socket that is already closed.

For in-process slot reuse, the usedSlots bit is only cleared in ReleaseDevice, so the same process will not hand the slot out early. But ReclaimLeaked (which, after an orchestrator restart, decides who is still connected purely from the pid file) and any bare device-probe path cannot rely on this predicate to prove the device has come to rest.

2. partscan noise (deployment layer)

The nbd module is loaded with max_part=16, so every /sys/block/nbdX/partscan == 1. As a result, both nbdnl.Connect's set_capacity and the os.Open(devicePath) in Open() trigger the kernel's automatic partition scan, issuing probe reads against sector 0 / RDB / LDM. Those reads land on connections that are either mid-teardown (socket already closed) or freshly established and not yet stable, producing the I/O error sector 0 and unable to read partition table lines.

The sandbox rootfs is a bare ext4 image with no partition table, so these scans are pure waste: they spam dmesg, burn backend I/O on the create hot path, and can drown out genuine EIO.

Confirmed runtime facts (dev-sandbox-2)

  • /sys/block/nbdX/inflight exists; idle value is 0 0 (two columns: in-flight reads / writes).
  • /sys/block/nbdX/holders/ exists; empty when nothing holds the device.
  • /sys/block/nbdX/partscan == 1, read-only (governed by module param max_part=16).
  • nbds_max == 4096.

Proposed fix

A. Strengthen isDeviceFree: on top of no pid + size==0, additionally require

  • inflight == "0 0" (no in-flight blk_mq requests), and
  • holders/ empty (no udev / partition-probe / mount reference).

Only when all hold is the device declared free. This actually proves quiescence and closes the window in which the async-stage probe reads occur. Both new signals are best-effort: if the sysfs files are absent (older kernels), the check falls back to the size-only decision so the pool never wedges on a signal that will never appear.

B. Suppress partscan (deployment): set options nbd nbds_max=4096 max_part=0 via /etc/modprobe.d/nbd.conf so nbd creates no partitions and triggers no scan.

The accompanying PR lands A (Go layer, with tests). B is a deployment-side recommendation given in the PR description.

Lingua principale
Go
Stelle
1.6k
Fork
438
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di e2b-dev/runtime

Tutte le issue di e2b-dev/runtime

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.