dendrite unresponsive in racktest.

Start with the captured log at /staff/core/dendrite-164 and the svc:/oxide/dendrite:default service status. Reproduce the sidecar and scrimlet reboot sequence, then inspect where the firmware-loading messages stop responding. Done means the cause is identified and dendrite responds after the documented restart or reboot sequence.

This is on a sidecar that has passed loopback testing but rebooting of the sidecar and scrimlet in that order return us to this same state where dendrite is running but not responding.

dendrite log was captured in /staff/core/dendrite-164

logs end like this:

BRM44220001-switch # tail -f $(svcs -L dendrite)
{"msg":"fw_load       RAM : 0x01000000","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:05.357697219Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"Done!\n","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:25.417407494Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"0: 32:0 : FW error: cmd=f000 : detail=0000 : rsp=f000","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026296111Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"0: 32 : FW load :  /opt/oxide/dendrite/sidecar/share/tofino_sds_fw/credo/firmware/tof2_A0_grp_0_7_serdes.bin","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026333262Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"fw_load Hash Code : 0xb0bbe0","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026344543Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"fw_load Date Code : 0x00004aa6","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026354251Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"fw_load  CRC Code : 0xd920","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026362577Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"fw_load    Length : 52164","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026370662Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"fw_load     Entry : 0x0100b204","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026379219Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}
{"msg":"fw_load       RAM : 0x01000000","v":0,"name":"dpd","level":20,"time":"1986-12-28T00:02:26.026387324Z","hostname":"BRM44220001-switch","pid":1480,"unit":"bf-sde","module":"Dvm"}

service was online:

BRM44220001-switch # svcs
STATE          STIME    FMRI
legacy_run      0:00:30 lrc:/etc/rc2_d/S20sysetup
legacy_run      0:00:30 lrc:/etc/rc2_d/S89PRESERVE
online          0:00:20 svc:/system/svc/restarter:default
online          0:00:20 svc:/system/early-manifest-import:default
online          0:00:21 svc:/network/ip-interface-management:default
online          0:00:21 svc:/system/filesystem/root:default
online          0:00:21 svc:/network/netcfg:default
online          0:00:21 svc:/network/datalink-management:default
online          0:00:21 svc:/network/loopback:default
online          0:00:21 svc:/system/boot-archive:default
online          0:00:21 svc:/system/filesystem/usr:default
online          0:00:21 svc:/network/physical:default
online          0:00:21 svc:/system/device/local:default
online          0:00:21 svc:/system/identity:node
online          0:00:21 svc:/milestone/devices:default
online          0:00:21 svc:/network/ipsec/ipsecalgs:default
online          0:00:21 svc:/system/device/audio:default
online          0:00:21 svc:/system/filesystem/minimal:default
online          0:00:21 svc:/system/rmtmpfiles:default
online          0:00:21 svc:/network/ipsec/policy:default
online          0:00:21 svc:/milestone/network:default
online          0:00:21 svc:/network/iptun:default
online          0:00:21 svc:/network/initial:default
online          0:00:21 svc:/system/pkgserv:default
online          0:00:21 svc:/network/netmask:default
online          0:00:21 svc:/system/hostid:default
online          0:00:21 svc:/system/logadm-upgrade:default
online          0:00:21 svc:/system/auditset:default
online          0:00:22 svc:/system/pfexec:default
online          0:00:22 svc:/system/fmd:default
online          0:00:24 svc:/system/rbac:default
online          0:00:24 svc:/oxide/zone-network-setup:default
online          0:00:24 svc:/oxide/switch_zone_setup:default
online          0:00:25 svc:/oxide/uplink:default
online          0:00:25 svc:/network/service:default
online          0:00:25 svc:/system/cryptosvc:default
online          0:00:25 svc:/system/identity:domain
online          0:00:25 svc:/system/keymap:default
online          0:00:25 svc:/system/name-service-cache:default
online          0:00:28 svc:/milestone/name-services:default
online          0:00:30 svc:/system/manifest-import:default
online          0:00:30 svc:/milestone/single-user:default
online          0:00:30 svc:/system/coreadm:default
online          0:00:30 svc:/milestone/sysconfig:default
online          0:00:30 svc:/system/utmp:default
online          0:00:30 svc:/system/filesystem/local:default
online          0:00:30 svc:/system/console-login:default
online          0:00:30 svc:/system/cron:default
online          0:00:30 svc:/system/system-log:default
online          0:00:30 svc:/system/boot-archive-update:default
online          0:00:30 svc:/system/update-man-index:default
online          0:00:30 svc:/network/shares/group:default
online          0:00:33 svc:/milestone/multi-user:default
online          0:00:33 svc:/network/ssh:default
online          0:00:33 svc:/network/routing-setup:default
online          0:00:33 svc:/system/boot-config:default
online          0:00:33 svc:/network/shares/group:zfs
online          0:00:35 svc:/milestone/multi-user-server:default
online          0:00:35 svc:/site/compliance/time:default
online          0:00:35 svc:/site/compliance/techport-beacon:default
online          0:00:35 svc:/network/routing/ndp:default
online          0:00:35 svc:/network/chrony:default
online          0:00:38 svc:/oxide/tfport:default
online          0:00:40 svc:/oxide/dendrite:default

We tried restarting dendrite service, ending up in the same spot, and also then powered both the scrimlet and sidecar off and brought them back online, ending up in the same spot.