netwatch never turns on noq-udp's Apple fast path, so iroh's `fast-apple-datapath` does nothing

Open
#272 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust

Research direction

Start with netwatch's SocketState::bind and the existing test around socket rebinding; the issue says both changes are prepared but does not name their files. Check how noq-udp exposes UdpSocketState::set_apple_fast_path() and how iroh forwards the feature. Done means the non-default netwatch feature enables the fast path after checking symbol resolution, including after rebind, and iroh forwards that feature; resolve the partial-send concern before deciding whether it is safe to enable.

Written by the indexing model from the issue text.

Description

Since noq picked up quinn's runtime opt-in (quinn-rs/quinn#2463, in noq via n0-computer/noq#606, released in noq-udp 1.0.0-rc.0), noq-udp only uses sendmsg_x/recvmsg_x after the socket owner calls UdpSocketState::set_apple_fast_path(). For iroh's IP transports the owner is netwatch (SocketState::bind), and it never calls it. So iroh's default fast-apple-datapath feature (n0-computer/iroh#3724) still compiles the private-API code in, but every datagram takes its own sendmsg/recvmsg on macOS and iOS.

#244 tried to fix this and was closed to wait for quinn-rs/quinn#2748: a partial sendmsg_x returns how many datagrams it sent, noq-udp ignores that, and the rest of the batch is dropped silently.

I measured what it is worth with the switch on (1.5 GB over iroh 1.4.1 on an M1 Pro, macOS 27.2, both ends in one process, 3-5 runs each):

path before fast path on
loopback 70-72 MB/s, 31.9 CPU s/GB 72-73 MB/s, 33.4 CPU s/GB
through a local UDP relay adding 20 ms 94-99 MB/s, 38.9 CPU s/GB 109-114 MB/s, 34.5 CPU s/GB

So +2% to +16% speed, and CPU from +5% to -11%. Smaller than I hoped: the time spent inside the send syscall hardly changed, so on macOS most of the kernel cost is per datagram, not per call. No partial sendmsg_x showed up in about 3 M calls, but that was loopback only, not a real NIC queue.

Proposal: a non-default netwatch feature fast-apple-datapath (forwards to noq-udp's) that turns the fast path on in SocketState::bind, so on every rebind too, after checking that both symbols resolve. iroh's feature then forwards to it. That's one function plus one test in netwatch and one line in iroh; I have both ready.

Questions:

  • Is this OK before the partial-send fix, or should it wait for quinn-rs/quinn#2748 (or a port of it to noq-udp)?
  • iroh turns the feature on by default. Should it stay that way now that it actually does something? The App Store risk is the same either way, because the symbol names are in the binary as soon as the feature is compiled in.
Details
  • Runs: SendMe's loopback_speed test (release, iroh 1.4.1, netwatch 0.19.3 patched by path). The relay case puts a tokio UDP relay between the two endpoints. That relay still sends and receives one datagram per call, and its CPU counts too.
  • sendmsg_x carried about 5.6 datagrams per call on average. recvmsg_x returned about 2.4 per call on loopback and 3.9 through the relay.
  • With batched sends but one-datagram receives (an experiment, not proposed): loopback went to 81 MB/s at 30.7 CPU s/GB, but through the relay it was no better than before (101-110 MB/s, 38.5 CPU s/GB). Batched receives cost kernel time on loopback and save it through the relay.
  • sample over 8 s of loopback: __sendmsg 1779 samples before, sendmsg_x 1725-1988 after.
  • Escape hatch at runtime: TransportConfig::enable_segmentation_offload(false) keeps sends at one datagram per call.
  • Earlier quinn discussion of App Store guideline 2.5.1 and the private calls: quinn-rs/quinn#1993. Crashes on old macOS that were fixed since: quinn-rs/quinn#2155 and quinn-rs/quinn#2214. Symbols are resolved with dlsym since quinn-rs/quinn#2571.
Dominant language
Rust
Stars
34
Forks
56
Avg merge
2d 4h
Merged PRs (30d)
8

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from n0-computer/net-tools

All issues in n0-computer/net-tools

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.