[QWRT / QSDK 5.4] 建议回移 PPPoE GRO(net-next 55a5d8fca836):附 5.4 补丁和 IPQ807x 实测数据

Open
#14,303 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
c

Research direction

The proposed backport changes drivers/net/ppp/pppoe.c; start by reviewing that file and the supplied patch against QSDK linux-ipq-5.4. Check the reported packet socket behavior and the NSS LSO issue for bridged PPPoE, including the suggested ndo_features_check handling in nss-dp. Done means the change works for PPPoE GRO while preserving the reported IPv4 and IPv6 traffic integrity and avoiding packet loss in supported forwarding paths.

Written by the indexing model from the issue text.

Description

LEDE 的 6.12 和 6.18 内核已经带上了 650-v7.2-net-pppoe-implement-GRO-GSO-support.patch。但 QWRT 的 IPQ807x 固件还在用 QSDK 5.4 内核,PPPoE 拨号的下行方向完全没有 GRO。我把这个补丁回移到了 5.4,在 IPQ807x 上实测有明显收益。下面附上补丁和数据,供参考。

现状(QWRT IPQ807x,5.4.213,NSS FW 12.2-156)

本机终结的流量(路由器自身下载、代理)走的是这条路径:
nss-dp → napi_gro_receive(eth0) → pppoe_rcv → ppp_input → netif_rx(pppoe-wan) → ip_rcv → tcp

  • 5.4 的 pppoe.ko 没有注册 ETH_P_PPP_SES offload,所以每个 1492 字节的段都要单独走一遍协议栈。另外 netif_rx 会再入一次 backlog,结果是 softnet 处理次数约等于线上包数的 2 倍。
  • NSS 固件把例外包交给主机时用的是完整的 Eth+PPPoE 帧(经 eth0),固件本身不做合包。n2h 描述符里也没有 MSS 或 gso 字段,所以这件事只能由主机来做。
补丁

这是上游提交的 5.4 回移,只改 drivers/net/ppp/pppoe.c,不碰 af_inet.c 和 ip6_offload.c。改动点写在补丁头里:

  • 用 skb_gro_header_fast/_hard/_slow 代替 skb_gro_header;
  • 用 call_gro_receive() 代替 INDIRECT_CALL_INET;
  • gro_find_*_by_type 外面加了 rcu_read_lock。

补丁可以干净地打到 QSDK linux-ipq-5.4 的 pppoe.c 上,用路由器的 .config 编译通过。

650-v7.2-net-pppoe-implement-GRO-GSO-support.patch(5.4 回移)
From 55a5d8fca8365312ed53bc93ae1af67ee35c2915 Mon Sep 17 00:00:00 2001
From: Felix Fietkau <[email protected]>
Date: Wed, 13 May 2026 09:33:48 +0800
Subject: [PATCH] net: pppoe: implement GRO/GSO support

Only handles packets where the pppoe header length field matches the exact
packet length. Significantly improves rx throughput.

Signed-off-by: Felix Fietkau <[email protected]>
Signed-off-by: Qingfang Deng <[email protected]>
Link: https://patch.msgid.link/[email protected]
Signed-off-by: Paolo Abeni <[email protected]>

[Backport to Linux 5.4 (QSDK linux-ipq-5.4):
 - no <net/gro.h> / skb_gro_header(): use skb_gro_header_fast/_hard/_slow
 - no INDIRECT_CALL_INET for these callbacks: use call_gro_receive() and a
   direct ->gro_complete() call, so af_inet.c/ip6_offload.c are untouched
 - take rcu_read_lock() around gro_find_*_by_type() lookups
 - ppp_skb_is_compressed_proto() does not exist in 5.4, nothing to adapt]
---
 drivers/net/ppp/pppoe.c | 176 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 175 insertions(+), 1 deletion(-)

--- a/drivers/net/ppp/pppoe.c
+++ b/drivers/net/ppp/pppoe.c
@@ -443,7 +443,7 @@
 	if (skb->len < len)
 		goto drop;
 
-	if (pskb_trim_rcsum(skb, len))
+	if (!skb_is_gso(skb) && pskb_trim_rcsum(skb, len))
 		goto drop;
 
 	ph = pppoe_hdr(skb);
@@ -1264,6 +1264,176 @@
 	.size = sizeof(struct pppoe_net),
 };
 
+/* PPPoE GRO/GSO, backported to 5.4 from net-next 55a5d8fca836
+ * ("net: pppoe: implement GRO/GSO support", Felix Fietkau / Qingfang Deng).
+ */
+static u16 compare_pppoe_header(const struct pppoe_hdr *phdr,
+				const struct pppoe_hdr *phdr2)
+{
+	__be16 proto = *(const __be16 *)(phdr + 1);
+	__be16 proto2 = *(const __be16 *)(phdr2 + 1);
+
+	return (__force u16)((phdr->sid ^ phdr2->sid) | (proto ^ proto2));
+}
+
+static __be16 pppoe_hdr_proto(const struct pppoe_hdr *phdr)
+{
+	__be16 proto = *(const __be16 *)(phdr + 1);
+
+	switch (proto) {
+	case cpu_to_be16(PPP_IP):
+		return cpu_to_be16(ETH_P_IP);
+#if IS_ENABLED(CONFIG_IPV6)
+	case cpu_to_be16(PPP_IPV6):
+		return cpu_to_be16(ETH_P_IPV6);
+#endif
+	default:
+		return 0;
+	}
+}
+
+static struct sk_buff *pppoe_gro_receive(struct list_head *head,
+					 struct sk_buff *skb)
+{
+	const struct packet_offload *ptype;
+	unsigned int hlen, off_pppoe;
+	const struct pppoe_hdr *phdr;
+	struct sk_buff *pp = NULL;
+	struct sk_buff *p;
+	int flush = 1;
+	__be16 type;
+
+	off_pppoe = skb_gro_offset(skb);
+	hlen = off_pppoe + PPPOE_SES_HLEN;
+	phdr = skb_gro_header_fast(skb, off_pppoe);
+	if (skb_gro_header_hard(skb, hlen)) {
+		phdr = skb_gro_header_slow(skb, hlen, off_pppoe);
+		if (unlikely(!phdr))
+			goto out;
+	}
+
+	/* filter for session packets (type:1, ver:1, code:0) */
+	if (*(const __be16 *)phdr != cpu_to_be16(0x1100))
+		goto out;
+
+	/* ignore packets with padding or invalid length */
+	if (skb_gro_len(skb) != be16_to_cpu(phdr->length) + sizeof(*phdr))
+		goto out;
+
+	type = pppoe_hdr_proto(phdr);
+	if (!type)
+		goto out;
+
+	rcu_read_lock();
+	ptype = gro_find_receive_by_type(type);
+	if (!ptype)
+		goto out_unlock;
+
+	flush = 0;
+
+	list_for_each_entry(p, head, list) {
+		const struct pppoe_hdr *phdr2;
+
+		if (!NAPI_GRO_CB(p)->same_flow)
+			continue;
+
+		phdr2 = (const struct pppoe_hdr *)(p->data + off_pppoe);
+		if (compare_pppoe_header(phdr, phdr2))
+			NAPI_GRO_CB(p)->same_flow = 0;
+	}
+
+	skb_gro_pull(skb, PPPOE_SES_HLEN);
+	skb_gro_postpull_rcsum(skb, phdr, PPPOE_SES_HLEN);
+
+	pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb);
+
+out_unlock:
+	rcu_read_unlock();
+out:
+	skb_gro_flush_final(skb, pp, flush);
+
+	return pp;
+}
+
+static int pppoe_gro_complete(struct sk_buff *skb, int nhoff)
+{
+	struct pppoe_hdr *phdr = (struct pppoe_hdr *)(skb->data + nhoff);
+	__be16 type = pppoe_hdr_proto(phdr);
+	struct packet_offload *ptype;
+	unsigned int len;
+	int err = -ENOENT;
+
+	rcu_read_lock();
+	ptype = gro_find_complete_by_type(type);
+	if (!ptype)
+		goto out;
+
+	len = skb->len - (nhoff + sizeof(*phdr));
+	len = min(len, 0xFFFFU);
+	phdr->length = cpu_to_be16(len);
+
+	err = ptype->callbacks.gro_complete(skb, nhoff + PPPOE_SES_HLEN);
+out:
+	rcu_read_unlock();
+	return err;
+}
+
+static struct sk_buff *pppoe_gso_segment(struct sk_buff *skb,
+					 netdev_features_t features)
+{
+	struct sk_buff *segs = ERR_PTR(-EINVAL);
+	struct packet_offload *ptype;
+	struct pppoe_hdr *phdr;
+	__be16 orig_type, type;
+	int len, nhoff;
+
+	skb_reset_network_header(skb);
+	nhoff = skb_network_header(skb) - skb_mac_header(skb);
+
+	if (unlikely(!pskb_may_pull(skb, PPPOE_SES_HLEN)))
+		goto out;
+
+	phdr = (struct pppoe_hdr *)skb_network_header(skb);
+	type = pppoe_hdr_proto(phdr);
+	rcu_read_lock();
+	ptype = gro_find_complete_by_type(type);
+	if (!ptype || !ptype->callbacks.gso_segment) {
+		rcu_read_unlock();
+		goto out;
+	}
+
+	orig_type = skb->protocol;
+	__skb_pull(skb, PPPOE_SES_HLEN);
+	features &= ~NETIF_F_GSO_SOFTWARE;
+	segs = ptype->callbacks.gso_segment(skb, features);
+	rcu_read_unlock();
+	if (IS_ERR_OR_NULL(segs))
+		goto out;
+
+	skb = segs;
+	do {
+		phdr = (struct pppoe_hdr *)(skb_mac_header(skb) + nhoff);
+		len = skb->len - (nhoff + sizeof(*phdr));
+		phdr->length = cpu_to_be16(len);
+		skb->network_header = (u8 *)phdr - skb->head;
+		skb->protocol = orig_type;
+		skb_reset_mac_len(skb);
+	} while ((skb = skb->next));
+
+out:
+	return segs;
+}
+
+static struct packet_offload pppoe_packet_offload __read_mostly = {
+	.type = cpu_to_be16(ETH_P_PPP_SES),
+	.priority = 20,
+	.callbacks = {
+		.gro_receive = pppoe_gro_receive,
+		.gro_complete = pppoe_gro_complete,
+		.gso_segment = pppoe_gso_segment,
+	},
+};
+
 static int __init pppoe_init(void)
 {
 	int err;
@@ -1280,6 +1450,8 @@
 	if (err)
 		goto out_unregister_pppoe_proto;
 
+	if (IS_ENABLED(CONFIG_INET))
+		dev_add_offload(&pppoe_packet_offload);
 	dev_add_pack(&pppoes_ptype);
 	dev_add_pack(&pppoed_ptype);
 	register_netdevice_notifier(&pppoe_notifier);
@@ -1299,6 +1471,8 @@
 	unregister_netdevice_notifier(&pppoe_notifier);
 	dev_remove_pack(&pppoed_ptype);
 	dev_remove_pack(&pppoes_ptype);
+	if (IS_ENABLED(CONFIG_INET))
+		dev_remove_offload(&pppoe_packet_offload);
 	unregister_pppox_proto(PX_PROTO_OE);
 	proto_unregister(&pppoe_sk_proto);
 	unregister_pernet_device(&pppoe_net_ops);

实际在设备上跑的是同一组函数做成的独立模块:只注册 offload,原有的 pppoe.ko 不动,所以不用重刷固件就能 insmod/rmmod。如果你更想要这种形式,我也可以提供。

测试

QEMU:用同一棵 QSDK 5.4 源码编测试内核,开 SKB_RECYCLER(含 MULTI_CPU),在两块 virtio 网卡之间跑真实的 PPPoE 会话(pppoe-server 加 pppd)。

  • IPv4 和 IPv6 各传 48 MB 随机文件,加载前、加载中、卸载后,sha256 全部一致。
  • 在流量进行中 insmod/rmmod 20 次,没有 WARN、BUG 或 Oops。

实机(IPQ8074,QWRT 5.4.213,PPPoE 拨号):

本机直连下载,每轮 2×30 MB,取两轮平均:

不开 GRO PPPoE GRO
交给 pppoe-wan 的 skb 约 4.5 万 约 1.65 万(合并 2.5–3 倍)
softnet 处理次数 / 线上包 2.0 约 0.74
回的 ACK 约 1.9 万 约 1.1 万
softirq CPU(jiffies,HZ=100,四核合计) 111 80(约 −28%)
总忙碌 CPU(usr+sys+irq+softirq) 308 253(约 −18%)

文件 md5 前后一致,softnet 计数中没有丢包或积压。

转发流量:在 LAN 客户端做直连下载(IPv4/IPv6),md5 全部正确。合并后的 GSO 包转发到 LAN 时走 NSS 的 LSO 分段,lso_rx 丢包为 0。

代理(LAN 客户端 → 路由器 sing-box/anytls → PPPoE):

测试方法:在路由器本机跑 4 路并行下载,每路 40 MB,路径是 curl → Passwall TPROXY → sing-box(anytls)→ PPPoE,对端是 Cloudflare HKG。两种状态各跑两轮。这个场景下四个 A53 核基本跑满,吞吐受 CPU 限制。

不开 GRO PPPoE GRO
4 路总吞吐 481 / 488 Mbit/s 564 / 584 Mbit/s(约 +18%)
交给 pppoe-wan 的 skb(约 13 万线上包) 约 13 万 约 1.75 万(合并约 7.5 倍)
softnet 处理次数 约 27 万 约 4.6 万(−83%)
softirq CPU(jiffies) 335 / 353 218 / 216(约 −37%)

样本只有两轮,绝对值仅供参考。CPU 跑满时,省下来的 softirq 时间基本都转化成了吞吐。

需要注意的两点
  1. 用户态读取 PPPoE 会话帧的程序会受影响。 GRO 合并出的超级帧会直接送到监听 0x8864 的 packet socket,例如用户态 pppoe、pty 模式的 pppoe-server、PPPoE 中继。这类程序按单帧处理,会丢数据。在 QEMU 里,用 pty 模式的 pppoe-server 做对端时,上传吞吐塌到原来的约 1/15。内核模式的 PPPoE 没有问题。

  2. NSS LSO 会丢弃 PPPoE 封装的 GSO 帧(NSS.FW.12.2-156,lso_rx 节点在 0x4006caa0 处的解析逻辑只认 0x0800 和 0x86dd,0x8864 走丢包分支)。正常的 PPPoE 拨号不会触发:GRO 后的包已经在 pppoe 层解封装,而 ppp 设备本身没有 GSO。但如果 eth0 被桥接,比如做 PPPoE 透传,合并后的 PPPoE 帧会在二层原样转发到开了 TSO 的 NSS 口,被 LSO 丢掉,透传拨号就会断流。

    建议在 nss-dp 的 ndo_features_check 里对 ETH_P_PPP_SES 清掉 NETIF_F_GSO_MASK,让内核自己做软件分段。

    LEDE qualcommax 6.18 加 NSS 的组合也许有同样的问题,不过我没有在 6.18 上验证过。

Dominant language
C
Stars
31.6k
Forks
19k
Avg merge
22h 16m
Merged PRs (30d)
8

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from coolsnowwolf/lede

All issues in coolsnowwolf/lede

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.