RoCE ring: distant ranks have no fabric path — forward 2-hop transfers via flow label 16383 instead of the CPU relay

Read helios/src/execution/roce_fabric.rs and helios/native/roce_proxy.c to trace route planning, queue pair setup, and the CPU relay; compare them with docs/roce-ring.md. The body gives a proposed flow label, opt-in flag, and reference implementation, but leaves scope and route-list behavior undecided. A useful completion criterion is an agreed design and validated forwarding behavior; the prototype reports no clean end-to-end run.

Summary

--roce-ring supports ring-cabled clusters, but distant ranks (2 hops apart) always go through the native CPU relay. There is no code path for letting the fabric forward those transfers, even when the intermediate ConnectX device already has hardware forwarding rules installed. This issue describes the gap, the mechanism that makes 2-hop forwarding possible, a reference implementation, and a concrete proposal.

Where the gap is

helios/src/execution/roce_fabric.rs:374 — plan_ring_routes reduces the endpoint table to rank ± 1 before planning:

let peers: Vec<_> = (0..world)
    .filter(|p| *p == rank || *p == (rank + 1) % world || *p == (rank + world - 1) % world)
    .collect();

Every other peer then gets an empty route vector, which propagates to helios/native/roce_proxy.c:

• :570 and :591 — if (p == c->rank || ring_relayed(c, p)) continue; — no queue pair is created for a distant rank
• :886 — if (ring_relayed(c, p)) return ring_post(...) — the transfer is posted into the neighbor's relay mailbox instead
• :238 — ring_relayed is the single predicate deciding all three

So the only supported path for a non-neighbor is the CPU relay, which the documentation describes as such (docs/roce-ring.md:4: "A native CPU proxy forwards transfers addressed to its non-neighbours", :56-57: rank 0 → rank 2 relays through rank 1).

That is a correct fallback. The gap is that it is the only option — the fabric is never given the chance, even when it is wired to do the job.

The mechanism that makes 2-hop forwarding possible

The switchless-ring deployments in the DGX Spark community do not relay in software. They let the sender's NIC tag the packets that must be forwarded and let the intermediate device rewrite and hairpin them in hardware. The tag is the RoCEv2 UDP source port, which mlx5 derives from the IPv6 flow label in the GRH:

flow_label 16383  →  UDP source port 65535

Setting rtr.ah_attr.grh.flow_label per peer in connect_qp is the entire tagging step. The intermediate device then matches eth_type 0x88b5 + src_mac + dst_mac with an offloaded flower rule (skip_sw), rewrites the EtherType back to 0x0800 and the destination MAC to the far node's port, and mirred-redirects out the sibling port. No CPU touches the forwarded packets, and the kernel forwards nothing.

Both halves are already present in Flare's design:

• The sender's flow_label is set in connect_qp — it is currently hardcoded to 0, but the per-peer plumbing is there.
• The CPU relay machinery (ring_post, mailboxes, ring_complete) can stay exactly as it is, as the fallback when routes are absent.

Reference implementation

knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4 implements this end to end for its own runtime, which is a fork of the same ring proxy:

• runtime/b12x/b12x/comm/roce_ring/_roce_proxy.c:512-524 — the flow-label assignment, with the comment explaining that the forwarding rules match UDP source port 65535 and that neighbor queue pairs keep flow label 0 so they do not match:

  rtr.ah_attr.grh.flow_label =
      (c->world == 4 && ((p - c->rank + c->world) % c->world) == 2)
          ? 16383
          : 0;
  

• docs/switchless-ring.md — the topology, the /32 diagonal routes, the skip_sw flower rules, the hairpin queue size, the warning that the marker rewrites every source-port-65535 packet on the device (hence neighbor QPs must use flow label 0), and ib_write_lat -d <dev> -x 3 --flow_label=16383 as an independent check.

The same convention is used by the sparkring tooling that ships the mesh units (mlx5-rdma-tx-rewrite-probe --source-port 65535 --replacement-ethertype 0x88b5).

Proposal

Three changes, none of which alter behavior for clusters that do not opt in:

1. Plan routes for distant ranks. plan_ring_routes already receives the full endpoint table, so it can derive the pair for a 2-hop peer without any configuration: the local device is this rank's device on the link to the carrying neighbor, and the remote device is the distant rank's device on its link to the same neighbor. same_link / interface_prefix already answer that. The pair should be expressed in a way both ends read identically (ordering by the two device numbers alone), and the two rails must be disjoint.

2. Tag those queue pairs. In connect_qp, set flow_label = 16383 when the peer is not a neighbor and forwarding is enabled. Neighbor queue pairs keep 0.

3. Opt in with a CLI flag. --roce-ring-forward (or similar) alongside --roce-ring, threaded through ServingOptions::rank_options so Orbit passes it to every rank. It cannot default to on: it only works when the intermediate devices have forwarding rules installed and the sender NICs have the tagging agent running, which is operator-side setup.

What I prototyped, and where I stopped

I implemented the three changes above against e5bd6a6 to check whether the approach holds up:

• plan_ring_routes derives 2-hop pairs with the constraint that the pair must be a hairpin on the intermediate device (the arriving port and the departing port belong to the same adapter, so the redirect stays inside one device), oriented symmetrically so both ends compute the same pairing from their own endpoint lists.
• ring_relayed split into ring_distant (the topology predicate) and ring_relayed (ring_distant && !ring_hw), so the relay path and the forwarding path share one source of truth.
• flow_label set per peer as described.
• cargo build --locked --release succeeds; cargo test --manifest-path helios/Cargo.toml --features runtime-package,server passes 885 tests, including the ring routing test rewritten to assert the new non-neighbor pairs.

On a 4-node ring the transport does come up with the new routes (Connect RoCE transport. 2 rails per neighbor (experimental forwarded ring)), which confirms the route derivation and the QP setup. I did not get a clean end-to-end run: the remaining blocker is deployment-specific — the tagging agent was only installed on a subset of the ports the plan selected, so the packets that needed forwarding were never tagged and the collective timed out waiting for the distant rank. That is configuration, not the design, but it means I cannot claim a validated result, and I would rather report the idea than submit a PR I cannot back with measurements.

Question

Is ring forwarding for distant ranks in scope for Flare, or is the CPU relay intended to be the permanent answer for ring cabling? If it is in scope, would a PR along the lines above be welcome, and would you want the flag to also accept an explicit route list for deployments whose tagging coverage does not match the automatically derived pairing?