nftables mode: 1.96 connmark rules in the ip/ip6 mangle tables break iptables-nft users (Cilium agent exits)

Open
  • Is something wrong?
#22,363 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, linux
Domain
networking

Research direction

Start by locating the nftables firewall code that adds AddConnmarkSaveRule and trace why it writes to the shared ip and ip6 mangle tables whenever netfilter is enabled. Check the 1.96 change and reproduce with nftables mode, then verify that iptables-nft can list mangle and Cilium starts. The report gives reproduction steps and suggests possible fixes, but the right behavior may require understanding firewall ownership and when the rules are needed.

Written by the indexing model from the issue text.

Description

bug cilium kubernetes
What is the issue?

I run tailscaled on Kubernetes nodes with Cilium as the CNI. Since 1.96, with the firewall in nftables mode, tailscaled adds connmark rules
(AddConnmarkSaveRule, the rp_filter workaround for exit nodes and subnet
routers) to the ip mangle and ip6 mangle tables, as native nftables
expressions. These are the same tables iptables-nft uses. It does this
whenever netfilter is on, including on nodes that are not exit nodes or
subnet routers.

What ends up on the host (nft list table ip mangle; ip6 is the same):

table ip mangle {
	chain PREROUTING {
		type filter hook prerouting priority mangle; policy accept;
		ct state established,related meta mark set ct mark & 0x0000ff00
	}

	chain OUTPUT {
		type filter hook output priority mangle; policy accept;
		ct state new meta mark & 0x0000ff00 != 0x00000000 ct mark set meta mark & 0x0000ff00
	}
}

iptables-nft does not accept native expressions in tables it manages.
Anything on the host that uses iptables-nft then can't touch mangle.
For Cilium (iptables 1.8.8 in its image), the agent exits:

level=warn msg="iptables table is not available on this system" error="unable to run 'iptables -t mangle -L -n' iptables command: exit status 1 stderr=\"iptables v1.8.8 (nf_tables): table `mangle' is incompatible, use 'nft' tool.\"" table=mangle
level=fatal msg="unable to run agent: failed to start: IPv6 is enabled, but the needed ip6tables tables are unavailable (try disabling IPv6 in Cilium or installing ip6tables and kernel modules: ip6_tables, ip6table_mangle, ip6table_raw, ip6table_filter)"

That leaves a Kubernetes node with no CNI. 1.92.x and 1.94.x are fine on the
same hosts. In iptables mode (TS_DEBUG_FIREWALL_MODE=iptables) 1.96.4
works too, because the same rules are written as xtables rules, which
iptables-nft can read.

Steps to reproduce
  1. A Linux host with no iptables binary, so tailscaled picks nftables mode
    (or set TS_DEBUG_FIREWALL_MODE=nftables).
  2. Run tailscaled 1.96.4 and bring it up.
  3. iptables-nft -t mangle -L -n → "table `mangle' is incompatible, use 'nft' tool."
  4. With Cilium as the CNI, the cilium-agent exits as above.
Suggested fix

One of:

  • In nftables mode, put the connmark rules in a table of tailscaled's own
    instead of ip mangle / ip6 mangle,
  • or add them only when they are needed (exit node / subnet router in use),
  • or provide an opt-out.
Are there any recent changes that introduced the issue?

1.96 (connmark save/restore rules in mangle).

OS

Linux

OS version

A minimal Linux distribution for Kubernetes nodes, kernel 6.18.60

Tailscale version

1.96.4

Other software

Cilium (agent image with iptables v1.8.8, nf_tables backend)

Bug report

N/A

Dominant language
Go
Stars
38.8k
Forks
3.4k
Avg merge
2d 7h
Merged PRs (30d)
194

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tailscale/tailscale

All issues in tailscale/tailscale

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.