berlin switch 0 not transmitting packets from host to SPs

Start with the management-network statistics for the VSC7448-to-Tofino port and run the provided ./xcvrtrace.d script against dendrite. Compare the transmitted requests with the missing SP responses and the reported interface state. Done means identifying and correcting the host-to-SP packet path so SP responses return and dendrite can finish starting.

This morning on berlin, I was trying to test upgrade on Berlin and ran into an issue where the switch zone on scrimlet14 apparently couldn't talk to any SPs, which prevented dendrite from starting up. I believe the sequence of events was:

• I saw that the sidecar0 SP (BRM23230003) was running an image without a VERS in its caboose
• I manually updated it to version 1.0.43 via pilot
• I issued a pilot sp cycle BRM23230003 (this was wrong, see the next couple steps)
• I issued a pilot sp cycle BRM42220023 to reboot scrimlet14
• After scrimlet14 came back, the switch zone had no connectivity. I realized I should have issued a reset of the sidecar to get it to take the update, not a cycle, but now I'm not sure whether the issue here was that or whether it was already in the wedged state (the subject of this issue).
• I issued an SP reset to the sidecar.
• I pilot sp cycle'd scrimlet14 again.

Once scrimlet14 came back, its switch zone still appears to be DOA. It has a subset of the expected interfaces:

root@oxz_switchnull:~# ipadm
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
lo0/v6            static   ok           ::1/128
oxBootstrap0/ll   addrconf ok           fe80::8:20ff:fe35:131c%oxBootstrap0/10
oxBootstrap0/bootstrap6 static ok       fdb0:a840:2504:110::2/64
oxControlService0/ll addrconf ok        fe80::8:20ff:fe0b:e0e1%oxControlService0/10
oxControlService0/omicron6 static ok    fd00:1122:3344:101::2/64
gimlet0/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet0/10
gimlet1/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet1/10
gimlet2/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet2/10
gimlet3/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet3/10
gimlet4/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet4/10
gimlet5/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet5/10
gimlet6/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet6/10
gimlet7/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet7/10
gimlet8/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet8/10
gimlet9/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet9/10
gimlet10/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet10/10
gimlet11/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet11/10
gimlet12/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet12/10
gimlet13/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet13/10
gimlet14/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet14/10
gimlet15/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet15/10
gimlet16/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet16/10
gimlet17/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet17/10
gimlet18/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet18/10
gimlet19/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet19/10
gimlet20/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet20/10
gimlet21/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet21/10
gimlet22/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet22/10
gimlet23/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet23/10
gimlet24/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet24/10
gimlet25/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet25/10
gimlet26/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet26/10
gimlet27/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet27/10
gimlet28/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet28/10
gimlet29/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet29/10
gimlet30/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet30/10
gimlet31/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%gimlet31/10
psc0/ll           addrconf ok           fe80::aa40:25ff:fe58:d9f0%psc0/10
psc1/ll           addrconf ok           fe80::aa40:25ff:fe58:d9f0%psc1/10
sidecar0/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%sidecar0/10
sidecar1/ll       addrconf ok           fe80::aa40:25ff:fe58:d9f0%sidecar1/10
techport0/ll      addrconf ok           fe80::aa40:25ff:fe58:d9f0%techport0/10
techport0/v6      static   ok           fdb1:a840:2504:110::1/10
techport0/ll      addrconf ok           fdb1:a840:2504:110:aa40:25ff:fe58:d9f0/64
techport1/ll      addrconf ok           fe80::aa40:25ff:fe58:d9f0%techport1/10
techport1/v6      static   ok           fdb2:a840:2504:110::1/10
techport1/ll      addrconf ok           fdb2:a840:2504:110:aa40:25ff:fe58:d9f0/64
tofino0/ll        addrconf ok           fe80::aa40:25ff:fe58:d9f0%tofino0/10
tfportint0_0/ll   addrconf ok           fe80::aa40:25ff:fe58:d9f0%tfportint0_0/10

but:

• MGS hasn't been able to talk to any SPs
• dendrite is stuck starting up, emitting these errors every few seconds:

15:24:12.821Z ERRO dpd: failed to send message within the retry limit
    limit = 3
    task = io
    unit = transceiver-controller
15:24:12.821Z ERRO dpd: failed to fetch MAC addresses from SP
    reason = MaxRetries(3)

@mkeeter and @Aaron-Hartwig had previously encountered this same issue and had some extra debugging things to look at. From the other scrimlet or from castle, we can ask about the management network stats for the port between the VSC7448 and the Tofino. It shows that the SP is transmitting, but has received nothing:

PortStatus(Ok(PortStatus { port: 49, cfg: PortConfig { mode: BaseKr, dev: (Dev10g, 0), serdes: (Serdes10g, 0) }, link_status: Up, phy_status: None, counters: PortCounters { rx: PacketCount { multicast: 0, unicast: 0, broadcast: 0 }, tx: PacketCount { multicast: 19896, unicast: 0, broadcast: 0 }, link_down_sticky: true, phy_link_down_sticky: false } }))

@bnaecker had provided this dtrace script to watch dendrite traffic:

#!/usr/sbin/dtrace -Zqs

xcvr_ctl$target:::message-sent
{
    peer = json(copyinstr(arg0), "ok");
    header = json(copyinstr(arg1), "ok");
    msg = json(copyinstr(arg2), "ok");
    printf("Sent message to %s\n", peer);
    printf("  header: %s\n", header);
    printf("  msg: %s\n", msg);
}

xcvr_ctl$target:::message-received
{
    peer = json(copyinstr(arg0), "ok");
    header = json(copyinstr(arg1), "ok");
    msg = json(copyinstr(arg2), "ok");
    body = json(msg, "body");
    printf("Recv message from %s\n", peer);
    printf("  header: %s\n", header);
    printf("  msg: %s\n", msg);
}

Running it shows dendrite making requests but getting no responses:

BRM42220023 # dtrace -s ./xcvrtrace.d -p 1899
dtrace: script './xcvrtrace.d' matched 2 probes
CPU     ID                    FUNCTION:NAME
 84   6865 _ZN22transceiver_controller6ioloop6IoLoop19send_protocol_error28_$u7b$$u7b$closure$u7d$$u7d$17h95725e36e6cc0f35E:message-sent Sent message to ff02::1de:2
  header: {"version":1,"message_id":2302,"message_kind":"HostRequest"}
  msg: {"version":5,"body":{"HostRequest":"MacAddrs"}}

110   6865 _ZN22transceiver_controller6ioloop6IoLoop19send_protocol_error28_$u7b$$u7b$closure$u7d$$u7d$17h95725e36e6cc0f35E:message-sent Sent message to ff02::1de:2
  header: {"version":1,"message_id":2302,"message_kind":"HostRequest"}
  msg: {"version":5,"body":{"HostRequest":"MacAddrs"}}

110   6865 _ZN22transceiver_controller6ioloop6IoLoop19send_protocol_error28_$u7b$$u7b$closure$u7d$$u7d$17h95725e36e6cc0f35E:message-sent Sent message to ff02::1de:2
  header: {"version":1,"message_id":2302,"message_kind":"HostRequest"}
  msg: {"version":5,"body":{"HostRequest":"MacAddrs"}}

103   6865 _ZN22transceiver_controller6ioloop6IoLoop19send_protocol_error28_$u7b$$u7b$closure$u7d$$u7d$17h95725e36e6cc0f35E:message-sent Sent message to ff02::1de:2
  header: {"version":1,"message_id":2303,"message_kind":"HostRequest"}
  msg: {"version":5,"body":{"HostRequest":"MacAddrs"}}

 86   6865 _ZN22transceiver_controller6ioloop6IoLoop19send_protocol_error28_$u7b$$u7b$closure$u7d$$u7d$17h95725e36e6cc0f35E:message-sent Sent message to ff02::1de:2
  header: {"version":1,"message_id":2303,"message_kind":"HostRequest"}
  msg: {"version":5,"body":{"HostRequest":"MacAddrs"}}

 86   6865 _ZN22transceiver_controller6ioloop6IoLoop19send_protocol_error28_$u7b$$u7b$closure$u7d$$u7d$17h95725e36e6cc0f35E:message-sent Sent message to ff02::1de:2
  header: {"version":1,"message_id":2303,"message_kind":"HostRequest"}
  msg: {"version":5,"body":{"HostRequest":"MacAddrs"}}