[QWRT / QSDK 5.4] 建议回移 PPPoE GRO(net-next 55a5d8fca836):附 5.4 补丁和 IPQ807x 实测数据
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- c
- Domain
- backend, networking
Research direction
The proposed backport changes drivers/net/ppp/pppoe.c; start by reviewing that file and the supplied patch against QSDK linux-ipq-5.4. Check the reported packet socket behavior and the NSS LSO issue for bridged PPPoE, including the suggested ndo_features_check handling in nss-dp. Done means the change works for PPPoE GRO while preserving the reported IPv4 and IPv6 traffic integrity and avoiding packet loss in supported forwarding paths.
Written by the indexing model from the issue text.
Description
LEDE 的 6.12 和 6.18 内核已经带上了 650-v7.2-net-pppoe-implement-GRO-GSO-support.patch。但 QWRT 的 IPQ807x 固件还在用 QSDK 5.4 内核,PPPoE 拨号的下行方向完全没有 GRO。我把这个补丁回移到了 5.4,在 IPQ807x 上实测有明显收益。下面附上补丁和数据,供参考。
现状(QWRT IPQ807x,5.4.213,NSS FW 12.2-156)
本机终结的流量(路由器自身下载、代理)走的是这条路径:
nss-dp → napi_gro_receive(eth0) → pppoe_rcv → ppp_input → netif_rx(pppoe-wan) → ip_rcv → tcp
- 5.4 的
pppoe.ko没有注册ETH_P_PPP_SESoffload,所以每个 1492 字节的段都要单独走一遍协议栈。另外netif_rx会再入一次 backlog,结果是 softnet 处理次数约等于线上包数的 2 倍。 - NSS 固件把例外包交给主机时用的是完整的 Eth+PPPoE 帧(经 eth0),固件本身不做合包。n2h 描述符里也没有 MSS 或 gso 字段,所以这件事只能由主机来做。
补丁
这是上游提交的 5.4 回移,只改 drivers/net/ppp/pppoe.c,不碰 af_inet.c 和 ip6_offload.c。改动点写在补丁头里:
- 用
skb_gro_header_fast/_hard/_slow代替skb_gro_header; - 用
call_gro_receive()代替INDIRECT_CALL_INET; gro_find_*_by_type外面加了rcu_read_lock。
补丁可以干净地打到 QSDK linux-ipq-5.4 的 pppoe.c 上,用路由器的 .config 编译通过。
650-v7.2-net-pppoe-implement-GRO-GSO-support.patch(5.4 回移)
From 55a5d8fca8365312ed53bc93ae1af67ee35c2915 Mon Sep 17 00:00:00 2001
From: Felix Fietkau <[email protected]>
Date: Wed, 13 May 2026 09:33:48 +0800
Subject: [PATCH] net: pppoe: implement GRO/GSO support
Only handles packets where the pppoe header length field matches the exact
packet length. Significantly improves rx throughput.
Signed-off-by: Felix Fietkau <[email protected]>
Signed-off-by: Qingfang Deng <[email protected]>
Link: https://patch.msgid.link/[email protected]
Signed-off-by: Paolo Abeni <[email protected]>
[Backport to Linux 5.4 (QSDK linux-ipq-5.4):
- no <net/gro.h> / skb_gro_header(): use skb_gro_header_fast/_hard/_slow
- no INDIRECT_CALL_INET for these callbacks: use call_gro_receive() and a
direct ->gro_complete() call, so af_inet.c/ip6_offload.c are untouched
- take rcu_read_lock() around gro_find_*_by_type() lookups
- ppp_skb_is_compressed_proto() does not exist in 5.4, nothing to adapt]
---
drivers/net/ppp/pppoe.c | 176 ++++++++++++++++++++++++++++++++++++++++-
1 file changed, 175 insertions(+), 1 deletion(-)
--- a/drivers/net/ppp/pppoe.c
+++ b/drivers/net/ppp/pppoe.c
@@ -443,7 +443,7 @@
if (skb->len < len)
goto drop;
- if (pskb_trim_rcsum(skb, len))
+ if (!skb_is_gso(skb) && pskb_trim_rcsum(skb, len))
goto drop;
ph = pppoe_hdr(skb);
@@ -1264,6 +1264,176 @@
.size = sizeof(struct pppoe_net),
};
+/* PPPoE GRO/GSO, backported to 5.4 from net-next 55a5d8fca836
+ * ("net: pppoe: implement GRO/GSO support", Felix Fietkau / Qingfang Deng).
+ */
+static u16 compare_pppoe_header(const struct pppoe_hdr *phdr,
+ const struct pppoe_hdr *phdr2)
+{
+ __be16 proto = *(const __be16 *)(phdr + 1);
+ __be16 proto2 = *(const __be16 *)(phdr2 + 1);
+
+ return (__force u16)((phdr->sid ^ phdr2->sid) | (proto ^ proto2));
+}
+
+static __be16 pppoe_hdr_proto(const struct pppoe_hdr *phdr)
+{
+ __be16 proto = *(const __be16 *)(phdr + 1);
+
+ switch (proto) {
+ case cpu_to_be16(PPP_IP):
+ return cpu_to_be16(ETH_P_IP);
+#if IS_ENABLED(CONFIG_IPV6)
+ case cpu_to_be16(PPP_IPV6):
+ return cpu_to_be16(ETH_P_IPV6);
+#endif
+ default:
+ return 0;
+ }
+}
+
+static struct sk_buff *pppoe_gro_receive(struct list_head *head,
+ struct sk_buff *skb)
+{
+ const struct packet_offload *ptype;
+ unsigned int hlen, off_pppoe;
+ const struct pppoe_hdr *phdr;
+ struct sk_buff *pp = NULL;
+ struct sk_buff *p;
+ int flush = 1;
+ __be16 type;
+
+ off_pppoe = skb_gro_offset(skb);
+ hlen = off_pppoe + PPPOE_SES_HLEN;
+ phdr = skb_gro_header_fast(skb, off_pppoe);
+ if (skb_gro_header_hard(skb, hlen)) {
+ phdr = skb_gro_header_slow(skb, hlen, off_pppoe);
+ if (unlikely(!phdr))
+ goto out;
+ }
+
+ /* filter for session packets (type:1, ver:1, code:0) */
+ if (*(const __be16 *)phdr != cpu_to_be16(0x1100))
+ goto out;
+
+ /* ignore packets with padding or invalid length */
+ if (skb_gro_len(skb) != be16_to_cpu(phdr->length) + sizeof(*phdr))
+ goto out;
+
+ type = pppoe_hdr_proto(phdr);
+ if (!type)
+ goto out;
+
+ rcu_read_lock();
+ ptype = gro_find_receive_by_type(type);
+ if (!ptype)
+ goto out_unlock;
+
+ flush = 0;
+
+ list_for_each_entry(p, head, list) {
+ const struct pppoe_hdr *phdr2;
+
+ if (!NAPI_GRO_CB(p)->same_flow)
+ continue;
+
+ phdr2 = (const struct pppoe_hdr *)(p->data + off_pppoe);
+ if (compare_pppoe_header(phdr, phdr2))
+ NAPI_GRO_CB(p)->same_flow = 0;
+ }
+
+ skb_gro_pull(skb, PPPOE_SES_HLEN);
+ skb_gro_postpull_rcsum(skb, phdr, PPPOE_SES_HLEN);
+
+ pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb);
+
+out_unlock:
+ rcu_read_unlock();
+out:
+ skb_gro_flush_final(skb, pp, flush);
+
+ return pp;
+}
+
+static int pppoe_gro_complete(struct sk_buff *skb, int nhoff)
+{
+ struct pppoe_hdr *phdr = (struct pppoe_hdr *)(skb->data + nhoff);
+ __be16 type = pppoe_hdr_proto(phdr);
+ struct packet_offload *ptype;
+ unsigned int len;
+ int err = -ENOENT;
+
+ rcu_read_lock();
+ ptype = gro_find_complete_by_type(type);
+ if (!ptype)
+ goto out;
+
+ len = skb->len - (nhoff + sizeof(*phdr));
+ len = min(len, 0xFFFFU);
+ phdr->length = cpu_to_be16(len);
+
+ err = ptype->callbacks.gro_complete(skb, nhoff + PPPOE_SES_HLEN);
+out:
+ rcu_read_unlock();
+ return err;
+}
+
+static struct sk_buff *pppoe_gso_segment(struct sk_buff *skb,
+ netdev_features_t features)
+{
+ struct sk_buff *segs = ERR_PTR(-EINVAL);
+ struct packet_offload *ptype;
+ struct pppoe_hdr *phdr;
+ __be16 orig_type, type;
+ int len, nhoff;
+
+ skb_reset_network_header(skb);
+ nhoff = skb_network_header(skb) - skb_mac_header(skb);
+
+ if (unlikely(!pskb_may_pull(skb, PPPOE_SES_HLEN)))
+ goto out;
+
+ phdr = (struct pppoe_hdr *)skb_network_header(skb);
+ type = pppoe_hdr_proto(phdr);
+ rcu_read_lock();
+ ptype = gro_find_complete_by_type(type);
+ if (!ptype || !ptype->callbacks.gso_segment) {
+ rcu_read_unlock();
+ goto out;
+ }
+
+ orig_type = skb->protocol;
+ __skb_pull(skb, PPPOE_SES_HLEN);
+ features &= ~NETIF_F_GSO_SOFTWARE;
+ segs = ptype->callbacks.gso_segment(skb, features);
+ rcu_read_unlock();
+ if (IS_ERR_OR_NULL(segs))
+ goto out;
+
+ skb = segs;
+ do {
+ phdr = (struct pppoe_hdr *)(skb_mac_header(skb) + nhoff);
+ len = skb->len - (nhoff + sizeof(*phdr));
+ phdr->length = cpu_to_be16(len);
+ skb->network_header = (u8 *)phdr - skb->head;
+ skb->protocol = orig_type;
+ skb_reset_mac_len(skb);
+ } while ((skb = skb->next));
+
+out:
+ return segs;
+}
+
+static struct packet_offload pppoe_packet_offload __read_mostly = {
+ .type = cpu_to_be16(ETH_P_PPP_SES),
+ .priority = 20,
+ .callbacks = {
+ .gro_receive = pppoe_gro_receive,
+ .gro_complete = pppoe_gro_complete,
+ .gso_segment = pppoe_gso_segment,
+ },
+};
+
static int __init pppoe_init(void)
{
int err;
@@ -1280,6 +1450,8 @@
if (err)
goto out_unregister_pppoe_proto;
+ if (IS_ENABLED(CONFIG_INET))
+ dev_add_offload(&pppoe_packet_offload);
dev_add_pack(&pppoes_ptype);
dev_add_pack(&pppoed_ptype);
register_netdevice_notifier(&pppoe_notifier);
@@ -1299,6 +1471,8 @@
unregister_netdevice_notifier(&pppoe_notifier);
dev_remove_pack(&pppoed_ptype);
dev_remove_pack(&pppoes_ptype);
+ if (IS_ENABLED(CONFIG_INET))
+ dev_remove_offload(&pppoe_packet_offload);
unregister_pppox_proto(PX_PROTO_OE);
proto_unregister(&pppoe_sk_proto);
unregister_pernet_device(&pppoe_net_ops);
实际在设备上跑的是同一组函数做成的独立模块:只注册 offload,原有的 pppoe.ko 不动,所以不用重刷固件就能 insmod/rmmod。如果你更想要这种形式,我也可以提供。
测试
QEMU:用同一棵 QSDK 5.4 源码编测试内核,开 SKB_RECYCLER(含 MULTI_CPU),在两块 virtio 网卡之间跑真实的 PPPoE 会话(pppoe-server 加 pppd)。
- IPv4 和 IPv6 各传 48 MB 随机文件,加载前、加载中、卸载后,sha256 全部一致。
- 在流量进行中 insmod/rmmod 20 次,没有 WARN、BUG 或 Oops。
实机(IPQ8074,QWRT 5.4.213,PPPoE 拨号):
本机直连下载,每轮 2×30 MB,取两轮平均:
| 不开 GRO | PPPoE GRO | |
|---|---|---|
| 交给 pppoe-wan 的 skb | 约 4.5 万 | 约 1.65 万(合并 2.5–3 倍) |
| softnet 处理次数 / 线上包 | 2.0 | 约 0.74 |
| 回的 ACK | 约 1.9 万 | 约 1.1 万 |
| softirq CPU(jiffies,HZ=100,四核合计) | 111 | 80(约 −28%) |
| 总忙碌 CPU(usr+sys+irq+softirq) | 308 | 253(约 −18%) |
文件 md5 前后一致,softnet 计数中没有丢包或积压。
转发流量:在 LAN 客户端做直连下载(IPv4/IPv6),md5 全部正确。合并后的 GSO 包转发到 LAN 时走 NSS 的 LSO 分段,lso_rx 丢包为 0。
代理(LAN 客户端 → 路由器 sing-box/anytls → PPPoE):
测试方法:在路由器本机跑 4 路并行下载,每路 40 MB,路径是 curl → Passwall TPROXY → sing-box(anytls)→ PPPoE,对端是 Cloudflare HKG。两种状态各跑两轮。这个场景下四个 A53 核基本跑满,吞吐受 CPU 限制。
| 不开 GRO | PPPoE GRO | |
|---|---|---|
| 4 路总吞吐 | 481 / 488 Mbit/s | 564 / 584 Mbit/s(约 +18%) |
| 交给 pppoe-wan 的 skb(约 13 万线上包) | 约 13 万 | 约 1.75 万(合并约 7.5 倍) |
| softnet 处理次数 | 约 27 万 | 约 4.6 万(−83%) |
| softirq CPU(jiffies) | 335 / 353 | 218 / 216(约 −37%) |
样本只有两轮,绝对值仅供参考。CPU 跑满时,省下来的 softirq 时间基本都转化成了吞吐。
需要注意的两点
-
用户态读取 PPPoE 会话帧的程序会受影响。 GRO 合并出的超级帧会直接送到监听 0x8864 的 packet socket,例如用户态
pppoe、pty 模式的pppoe-server、PPPoE 中继。这类程序按单帧处理,会丢数据。在 QEMU 里,用 pty 模式的 pppoe-server 做对端时,上传吞吐塌到原来的约 1/15。内核模式的 PPPoE 没有问题。 -
NSS LSO 会丢弃 PPPoE 封装的 GSO 帧(NSS.FW.12.2-156,
lso_rx节点在 0x4006caa0 处的解析逻辑只认 0x0800 和 0x86dd,0x8864 走丢包分支)。正常的 PPPoE 拨号不会触发:GRO 后的包已经在 pppoe 层解封装,而 ppp 设备本身没有 GSO。但如果 eth0 被桥接,比如做 PPPoE 透传,合并后的 PPPoE 帧会在二层原样转发到开了 TSO 的 NSS 口,被 LSO 丢掉,透传拨号就会断流。建议在 nss-dp 的
ndo_features_check里对ETH_P_PPP_SES清掉NETIF_F_GSO_MASK,让内核自己做软件分段。LEDE qualcommax 6.18 加 NSS 的组合也许有同样的问题,不过我没有在 6.18 上验证过。
- Dominant language
- C
- Stars
- 31.6k
- Forks
- 19k
- Avg merge
- 22h 16m
- Merged PRs (30d)
- 8
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from coolsnowwolf/lede
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
coolsnowwolf/lede#14305 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
coolsnowwolf/lede#14304 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 48/100
coolsnowwolf/lede#14295 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 Over a week Newbie friendliness 12/100
coolsnowwolf/lede#14289 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
coolsnowwolf/lede#14287 ·
Maintainers usually reply within 1 day
All issues in coolsnowwolf/lede
Similar issues
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
389ds/389-ds-base#8157 ·
Maintainers usually reply within 2 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Needs-Triage
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 2 days