[Bug]: e1000 (82540EM) guest TX queue hangs ~5 s at random ("NETDEV WATCHDOG ... Reset adapter") on 7.1.x Linux host, bridged; virtio-net alternative breaks permanently ("is not a head!")

Aperta
  • Qualcosa non va?
#975 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c, linux
Ambito
infrastructure, networking

Direzione di ricerca

Start by reviewing the attached VBox logs and guest kernel excerpts alongside the e1000 and virtio-net network emulation code; the report names neither source files nor a deterministic reproducer. First investigate how to capture a useful trace across the intermittent TX stalls, then compare the failure evidence with the reported adapter resets. Done means identifying the cause or providing diagnostic guidance for reproducing and tracing it.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Version

7.1.12

Host OS Type

Linux

Host OS name + version

Debian GNU/Linux 11 (bullseye), kernel 5.10.0-35-amd64

Host Architecture

x86

Guest OS Type

Linux

Guest Architecture

x86

Guest OS name + version

Debian GNU/Linux 12 (bookworm), kernel 6.1.0-34-amd64, 4 vCPU, 8 GB RAM, old Guest Additions (VBoxService 5.2.42, module reports 6.0.0)

Component

Network

What happened?
Version

7.1.14 r170994 (problem first seen after upgrading from 5.2.x to 7.1.10 in July 2025)

Host OS Type

Linux

Host OS name + version

Debian GNU/Linux 11 (bullseye), kernel 5.10.0-35-amd64

  • Host A, B: 2x Xeon Gold 6248R (96 threads), 754 GB RAM, 25-30 running VMs, NIC Intel X710 (i40e) 10G
  • Host C: 2x Xeon E5-2697 v2 (48 threads), 377 GB RAM, 5 running VMs, NIC Intel I350 (igb) 1G
  • Networking: bond0 (active-backup / 802.3ad) -> VLAN bond0. -> Linux bridge -> VM in Bridged mode
Host Architecture

x86

Guest OS Type

Linux

Guest Architecture

x86

Guest OS name + version

Debian GNU/Linux 12 (bookworm), kernel 6.1.0-34-amd64, 4 vCPU, 8 GB RAM, old Guest Additions (VBoxService 5.2.42, module reports 6.0.0)

Component

Network

What happened?

This is a test system: a 3-node MariaDB 11.4 Galera cluster, one VM per host.
Each VM has a bridged Intel PRO/1000 MT Desktop (82540EM) adapter.

The guest e1000 adapter periodically stops transmitting. After ~5 s the guest
kernel watchdog resets it:

e1000 0000:00:03.0 enp0s3: NETDEV WATCHDOG: CPU: 2: transmit queue 0 timed out 5152 ms
e1000 0000:00:03.0 enp0s3: Reset adapter
e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

A 5 s TX stall is longer than the Galera timeouts (peer 3 s / suspect 5 s),
so the node is evicted from the cluster and rejoins a few seconds later.
keepalived also loses the link and moves the VIP.

The problem is spread out in time and cannot be reproduced on demand.
Resets happen at random: any hour of the day, no link to cron/backup/load
peaks, and the interval between resets ranges from 3 minutes to 9.5 days
(median ~1 day on the busiest node). There is no workload that triggers it.

Statistics (guest kernel logs; full timestamp lists are attached as CSV):

Node Host Period Resets Per month TX stall (median / max)
DB-1 A (96 thr, ~25 VMs) 2025-11-24 .. 2026-11-12 235 5-33 5.4 s / 6.0 s
DB-2 B (96 thr, ~30 VMs) 2025-10-23 .. 2026-11-12 165 4-23 5.4 s / ~6 s
DB-3 C (48 thr, 5 VMs) 2025-02-24 .. 2026-11-12 80 1-23 ~5 s
  • Busier hosts see more resets, but the hosts are not saturated: CPU load ~12 %,
    VM vCPU scheduler wait ~0.2 %, disk await < 1 ms, network ~0.1 % of 10G.
  • It is per VM, not a host event. Two VMs on the same host (both e1000) reset
    independently over one year: 0 coincidences within ±60 s.
  • It started with the VirtualBox upgrade. On another VM with a continuous kernel log
    there were 0 resets in 7.5 months on 5.2.34, and the first reset came 11 h after the
    upgrade to 7.1.10. DB-3 had no resets from 2025-02 until the host upgrade
    (2025-07-15); the first one was on 2025-08-13.
  • Our production Galera cluster (3 more VMs on other hosts of the same kind) shows
    the same pattern: 13-36 resets/month per node.

Related observations:

  1. TSO. With TSO enabled in the guest, upload from the VM collapses
    (iperf3: 2 Mbit/s up vs 929 Mbit/s down). Since then TSO has been disabled
    (ethtool -K enp0s3 tso off), which restores 918 Mbit/s. This looks like #801.
    The TX resets continue with TSO off.

  2. virtio-net as an alternative. We switched one node to virtio-net with TSO on:
    937/931 Mbit/s. After ~5 days the guest logged

    virtio_net virtio0: output.0:id 146 is not a head!
net enp0s3: Unexpected TXQ (0) queue failure: -5   (x 101 610)

    The TX virtqueue stayed broken until the VM was rebooted, so the node had no network
    for 14 h. The link stayed "up" and RX kept working. We reverted to e1000.
    The same message was reported on the forum in 2017 (t=82524) and in ticket #11455.
    VBox.log of that run is attached; it has nothing at the failure time.

  3. VBox.log shows nothing. VBox.log contains nothing about either failure.
    Example: the attached e1000 log of DB-3 starts 2025-11-30 08:50 (guest local time, UTC+2).
    The guest reset at 15:13:43 the same day (~6 h 23 min into the log) left no line there.
    The release log then reaches its size cap after ~16 days, so the remaining 17 resets
    of that run (listed in the guest excerpt) were not logged at all. The only data are the
    device statistics at power-off (E1000#0 "Lost interrupts: 0"; virtio-net has no error counters).

Impact: every eviction triggers Zabbix alerts, so the on-call team gets flooded with
SMS several times a day across the test and production clusters.

How can we reproduce this?

No deterministic reproducer. Run a Linux guest with a bridged 82540EM adapter under normal,
moderate network load (Galera replication, ~0.1-1 MB/s) for several days on a VirtualBox 7.1.x
Linux host with several other VMs. NETDEV WATCHDOG resets appear at random, roughly once a day.

Questions
  1. Is this a known issue in the e1000 (and virtio-net) emulation in 7.x on Linux hosts?
    Is it fixed in 7.1.x / 7.2.x?
  2. Which logging or debug settings (release log groups, extradata, statistics) should we enable
    to capture a useful trace the next time it happens? We can keep it on for several days.
  3. Is there any device or extradata setting (TX queue/timers, interrupt moderation,
    another adapter type) that you recommend as a workaround in the meantime?

vbox-bug-logs.zip

How can we reproduce this?

The problem is spread out in time and cannot be reproduced on demand

Did you upload all of your necessary log files, screenshots, etc.?
  • Yes, I've uploaded all pertinent files to this issue.
Lingua principale
C
Stelle
1.4k
Fork
219
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di VirtualBox/virtualbox

Tutte le issue di VirtualBox/virtualbox

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.