Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

dpd wedges during bring-up, Unhandled FIFO 36 error, API never binds

Open
  • Is something wrong?
#298 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Quiet
Tech stack
rust
Domain
api, networking

Research direction

No source file or test is identified in the issue. Start by reading the attached pstack and log, then trace dpd's SDE bring-up around the pipe_mgr_drv_completion_cb assertion and the point where the :12224 API should bind. Done means dpd completes initialization, the API listens on :12224, swadm connects, and BGP sessions establish without the FIFO 36 assertion.

Written by the indexing model from the issue text.

Description

After a rack reboot (power outage on Madrid), dpd on sled 16's switch zone comes up but never finishes initialization: the dropshot API (:12224) is never bound, so swadm gets connection-refused and BGP establish fails. On svcadm restart, dpd gets further into SDE bring-up and then hits a bf-sde assertion in pipe_mgr_drv_completion_cb ("Unhandled FIFO 36"), and the API still never binds. The peer switch (oxz_switch0) came back from the same reboot healthy.

Symptom 1: initial wedge (post-reboot, before any restart)

  • svcs shows svc:/oxide/dendrite:default online, dpd PID alive.
  • No listener on :12224 (netstat shows dpd owns only AF_UNIX sockets + the SMF log fd
  • All tokio runtime workers parked
  • SMF log /var/svc/log/oxide-dendrite:default.log was 0 bytes
  • swadm Connection refused (os error 146) on localhost:12224.
  • All BGP sessions on this switch stuck in Connect/Peer ASN: None
/opt/oxide/dendrite/bin/swadm switch-port transceiver monitors qsfp15
Error: failed to get transceiver monitors

Caused by:
    0: Communication Error: error sending request for url (http://localhost:12224/ports/qsfp15/transceiver/monitors)
    1: error sending request for url (http://localhost:12224/ports/qsfp15/transceiver/monitors)
    2: client error (Connect)
    3: tcp connect error
    4: Connection refused (os error 146)

Symptom 2: after svcadm restart svc:/oxide/dendrite:default

dpd now logs and progresses through SDE bring-up, but then:

Unhandled FIFO 36/opType 0 MsgId 0x0 at pipe_mgr_drv_completion_cb:3983.
ASSERTION FAILED: "0" (0) from pipe_mgr_drv_completion_cb:3984

The :12224 API still never binds after the restart.

pstack summary:

  • bf-sde threads (bf_dma, bf_interrupt, bf_port_fsm, bf_switchd_process_async_*) alive in their normal poll/usleep loops.
  • Every tokio-runtime-worker parked in park_condvar.
  • dpd holds no TCP listeners; only AF_UNIX sockets + the log fd.

(full pstack + log attached)

pstack_madrid_16_dpd.txt

Dominant language
Rust
Stars
21
Forks
3
Avg merge
10h 45m
Merged PRs (30d)
2

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oxidecomputer/dendrite

All issues in oxidecomputer/dendrite

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.