macOS: while the VM runs, host traffic loses the DF bit (pf 'scrub ... no-df' on the primary NIC), breaking large TLS 1.3 ClientHellos on some networks

Open
#350 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
macos, rust
Domain
networking

Research direction

Start by tracing Kiri’s macOS VM networking setup and how it attaches Virtualization.framework NAT; the issue does not name source files or tests. Investigate whether the host-wide pf scrub rule can be avoided or whether an alternative networking mode is feasible. Done means the VM can run without changing the DF bit on host traffic, with a regression check for the reported TLS handshake failure.

Written by the indexing model from the issue text.

Description

Kiri version and system
  • Kiri 0.2.1 (/Applications/Kiri.app, daemon via ~/Library/LaunchAgents/dev.kiri.daemon.plist)
  • macOS 27.0.1, Apple M4 Pro
  • Host network: USB Ethernet (AX88179B) or built-in Wi-Fi, MTU 1500, behind a corporate firewall
What you did
  1. Started Kiri normally (the daemon boots its VM with Virtualization.framework NAT; bridge100 comes up as 192.168.64.1/24).
  2. On the host (not inside a container), opened a TLS 1.3 connection with a default OpenSSL 3.5+/Node client, which sends a post-quantum ClientHello (X25519MLKEM768 + X25519 key shares, ~1.47 KB, so it spans two TCP segments):
echo | openssl s_client -connect api.anthropic.com:443 -servername api.anthropic.com -tls1_3
What happened

Every attempt failed with write:errno=54 (ECONNRESET). This affects Node apps such as Claude Code as well. The same handshake with -groups X25519 (small ClientHello) succeeded.

While Kiri is running, macOS InternetSharing loads this pf anchor for the VM's NAT:

# pfctl -a com.apple.internet-sharing/shared_v4 -sr -sn
nat on en8 inet from 192.168.64.0/24 to any -> (en8:0) extfilter ei
scrub on en8 all no-df fragment reassemble
scrub on bridge100 all no-df max-mss 1460 fragment reassemble
...

scrub on <primary NIC> all no-df clears the DF bit on all host traffic leaving the primary interface, not only on VM traffic. A packet capture on the host NIC showed:

  • The outgoing full-size segment (1500 bytes) had no DF bit and IP ID 0.
  • The server SACKed only the small trailing segment, so the 1500-byte segment never arrived.
  • The server then sent RST.

On this network path, large non-DF packets are dropped. Hosts without the rule (another Mac on the same LAN) are unaffected. The rule also follows the primary interface: after switching from Ethernet to Wi-Fi, the failure moved to Wi-Fi as well.

After stopping Kiri (launchctl bootout gui/$UID/dev.kiri.daemon), the VM and bridge100 went away and the same handshake succeeded 5/5 times on the same network. Node also negotiated TLS 1.3 again.

What you expected instead

Running Kiri should not change how the host's own traffic is sent. Host packets should keep their DF bit (Path MTU Discovery) whether or not the Kiri VM is up.

Anything else
  • The no-df scrub rule appears to be added by macOS InternetSharing for Virtualization.framework's NAT attachment, so Kiri may not control it directly. Possible directions:
    • A networking mode that does not rely on the shared NAT. Examples: a userspace network stack over VZFileHandleNetworkDeviceAttachment (gvproxy/vfkit style), or vmnet in a mode that does not install host-wide scrub rules.
    • At minimum, documenting this side effect and the workaround.
  • Workarounds that helped:
    • Stopping Kiri.
    • Using a different upstream network (a phone hotspot).
    • Lowering the host NIC MTU (for example networksetup -setMTU <service> 1400), so host packets never reach 1500 bytes.
  • Reproducible every time on the affected network, and only while the Kiri VM is running.
Dominant language
Rust
Stars
7.5k
Forks
329
Avg merge
7h 48m
Merged PRs (30d)
105

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from libracoder/kiri

All issues in libracoder/kiri

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.