Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Faulted link observed during MUPdate with limited retrospective debugging

Aperta
  • Qualcosa non va?
#253 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

@rcgoodfellow ci sta già lavorando.

Dal 1/4/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

We observed failures during initial manufacturing MUPdate of sled 22 in a rack. I'm leaving the issue title here open ended since I am relatively unfamiliar with the system. The things we'd like are (1) a mechanism to understand why a link (e.g. rear22/0) is in a Faulted state and (2) a mechanism (e.g. logs) to get this retrospectively - for example, after ignition cycle when links are back up.

See internal ticket for details here: https://github.com/oxidecomputer/mfg-troubleshooting/issues/926.

/staff/adam/dendrite-fault-rear22/ contains the output of swadm link history and the dendrite service logs.

adam@atrium ~ $ find /staff/adam/dendrite-fault-rear22/ -type f
/staff/adam/dendrite-fault-rear22/XXXXXXXX/swadm-link-history-rear220.txt
/staff/adam/dendrite-fault-rear22/XXXXXXXX/oxide-dendrite:default.log
/staff/adam/dendrite-fault-rear22/YYYYYYYY/oxide-dendrite:default.log
/staff/adam/dendrite-fault-rear22/YYYYYYYY/swadm-link-history-rear220.txt

MUPdate was hung downloading installinator because both links were down:

XXXXXXXX # dladm
LINK        CLASS     MTU    STATE    BRIDGE     OVER
cxgbe0      phys      1500   down     --         --
cxgbe1      phys      1500   down     --         --
bootstrap\_stub0 etherstub 9000 up     --         --
bootstrap0  vnic      1500   up       --         bootstrap\_stub0
XXXXXXXX #

swadm link ls in the switch zones showed rear22/0 in a Faulted state.

support@oxz_switch0:~$ swadm link ls
...
rear22/0   Copper  100G   RS    true     true     Faulted  xx:xx:xx:xx:xx:xx
...

We ignition cycled multiple times and XXXXXXXX / 22 rebooted with all links up:

XXXXXXXX # dladm
LINK        CLASS     MTU    STATE    BRIDGE     OVER
cxgbe0      phys      9000   up       --         --
cxgbe1      phys      9000   up       --         --
XXXXXXXX #

Likewise, we lost the Faulted state in swadm link ls:

support@oxz_switch0:~$ swadm link ls
...
rear22/0   Copper  100G   RS    true     true     Up       xx:xx:xx:xx:xx:xx
...

dendrite logs show the link transitioning out of Faulted:

20:17:41.822Z DEBG dpd: Link update
    link_id = 0
    new = Up
    old = Faulted
    port_id = rear22
    state = LinkUp
    unit = callback_handler

swadm link history showed loops of:

14241983  PortFSM  Idle                         -
14242053  PortFSM  Abort                        -
14247600  PortFSM  WaitAutoNegLinkTrainingDone  -
14250248  PortFSM  WaitAutoNegDone              -

until ignition cycles resulted in the link going up:

Time      Class    Subclass                     Channel  Details
3746629   PortFSM  LinkUp                       -
3746639   PortFSM  LinkDown                     -
3747009   PortFSM  WaitAutoNegLinkTrainingDone  -
4197483   PortFSM  WaitAutoNegDone              -
4198486   PortFSM  Idle                         -
Lingua principale
Rust
Stelle
20
Fork
3
Merge medio
8h 29m
PR unite (30g)
2

Preparare l'ambiente

Non abbiamo ancora controllato i file di configurazione di questo progetto. Parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di oxidecomputer/dendrite

Tutte le issue di oxidecomputer/dendrite

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.