Lost network wakeup

Aperta
  • Qualcosa non va?
#2,729 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
networking, operating-systems

Direzione di ricerca

Start with the network task retry path in src/executor/network.rs around line 254, the sleep decision in src/executor/mod.rs around line 100, and wake_network_waker() in src/fd/socket/udp.rs around line 47. Reproduce the race with the provided cargo xtask ci rs command using --smp 2, then trace wakeups across cores. Done means the network task is not stranded before wfi and socket operations can wake an idle boot core.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem 1: Wakeup lost when NIC lock held

What normally happens when a packet arrives:

  1. The network card raises a interrupt to core 0.
  2. The interrupt handler marks the network async task as runnable.
  3. The async task is executed on core 0, takes the NIC lock and processes packets.

Race condition:

  1. Core 1 keeps grabbing the NIC lock, e.g. for accepting new connection for the web server.
  2. Core 0: Packet arrives. Network task runs.
  3. try_lock fails. The async task does not wait, but put's itself back in the queue.
    https://github.com/hermit-os/kernel/blob/caa649b10e61fdf1c77ad0f1d429713953ae48f8/src/executor/network.rs#L254
  4. executor::run tries to run the task up to 3 times. The scheduler only checks if any threads are ready to run, not if any async tasks are ready to run. If no tasks are ready it executes wfi.
    https://github.com/hermit-os/kernel/blob/caa649b10e61fdf1c77ad0f1d429713953ae48f8/src/executor/mod.rs#L100
  5. The network stack's async task is still pending.
  6. Nothing happens on core 0 till the next interrupt (e.g. issued by tcp retransmission).

This was very sporadic, but with 2b7a7cc08cb92f9bd6e4b869425f3bb152772871, the issue becomes easily reproducible.
The commit optimised the arming of a oneshot timers. The devicetree is not parsed for every call, but only once at startup.

cargo xtask ci rs --arch riscv64 --package httpd --smp 2 \
  --features ci,hermit/dhcpv4,hermit/tcp,hermit/virtio-net,hermit/loader \
  qemu --devices virtio-net-pci
...
Now listening on port 9975
[    1.567706][0][INFO  network   ] DHCP config acquired!
[    1.569874][0][INFO  network   ] IP address:   192.168.76.9/24
[    1.572302][0][INFO  network   ] Gateway:      192.168.76.2
[    1.574774][0][INFO  network   ] DNS server 0: 192.168.76.3
[CI] GET http://127.0.0.1:9975
Error: timeout: global
qemu-system-riscv64: terminating on signal 15 from pid 95011 (<unknown process>)

full_log.txt

Potential fix: Before going to sleep (wfi) async tasks that are ready, should be executed.

Problem 2: Missing IPI from cores trying to wake the network task

Socket operations can call wake_network_waker() to wakeup the network stack's async task.
https://github.com/hermit-os/kernel/blob/caa649b10e61fdf1c77ad0f1d429713953ae48f8/src/fd/socket/udp.rs#L47
The network stack's async task resides on the boot core, but socket operations can be issued by other cores too.
If the boot core is idle, the core, issuing the socket operation, should wake the boot core via an IPI.

Lingua principale
Rust
Stelle
1.5k
Fork
135
Merge medio
13h 8m
PR unite (30g)
8

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di hermit-os/kernel

Tutte le issue di hermit-os/kernel

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.