[RDMA] 如何让请求 attachment 直接落入满足 NVMe PRP 要求的缓冲区?
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- cpp
- Domain
- backend-api-design, networking
Research direction
The issue points to RdmaEndpoint::PostRecv() and DoPostRecv() as the receive path, and to IOReserveAlignedBuf in #993 and #2942 as related work. Read those implementations and discussions first, then determine whether the current API supports application supplied DMA buffers or constrained receive segments. Done means identifying an existing approach or defining and validating a feasible design for PRP compatible payloads; the issue reports no end to end SPDK write test.
Written by the indexing model from the issue text.
Description
我们在评估用 bRPC 1.17 RDMA + SPDK NVMe driver 做存储服务,希望网卡收到的 payload 能直接提交给 NVMe,省掉从 attachment 到磁盘 I/O 缓冲区的 CPU 拷贝。
写入的磁盘 offset 和长度始终按 LBA 对齐,不涉及 read-modify-write。我们用 nvme id-ctrl -H 确认测试盘(Samsung PM9A3)报告 SGLS = 0,所以 NVMe 侧需要用 PRP 表示数据布局。
从源码看,RdmaEndpoint::PostRecv() 预先分配固定大小的 IOBuf 接收块,DoPostRecv() 每次提交一个接收 SGE。解析 RPC 后,attachment 可能从块内某个偏移开始,并跨多个接收块。即使这些块来自网卡和 NVMe 都能访问的 DMA 内存池,块间也可能在 NVMe 内存页中间跳到不连续的 DMA 地址;如果这个分段点也不在 LBA 边界,就无法在那里拆成两条写命令。
看到 #993 和 #2942 中的 IOReserveAlignedBuf 可以主动预留对齐空间,但没找到用它指定 incoming attachment 接收位置的接口。目前只核对了源码和控制器能力,还没有做端到端 SPDK 写盘验证。
想请教:
- bRPC 目前是否有办法把 RDMA attachment 直接接收到应用提供的 DMA buffer,或约束接收分段,让 payload 满足 PRP 布局?
- 如果没有,bRPC 是否适合提供一个可选模式,把 RPC 元数据与按页对齐的 payload 分开接收?还是建议由 bRPC 传控制信息,另用单边 RDMA 把 payload 放到应用预分配的 buffer?
如果已有相关实现或讨论,麻烦指个方向,谢谢!
- Dominant language
- C++
- Stars
- 17.6k
- Forks
- 4.2k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 72
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/brpc
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 2 days
-
endpoint2hostname() prints a byte-swapped port for IPv6 endpointsPossibly taken @0z5a claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 2 days
-
perf(butil): ARM64 (aarch64) performance and modernisation improvements across atomics, CRC32C, spin-wait hints, and fast timersPossibly taken @jimmystewpot claimed this 2 days ago. Open
Difficulty 5/5 Over a week Newbie friendliness 12/100
Maintainers usually reply within 2 days
-
file:// naming service takes a comment after the address as the tagPossibly taken @0z5a claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
Maintainers usually reply within 2 days
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 20/100
apache/brpc#3707 · 3 comments ·
Maintainers usually reply within 2 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
protocolbuffers/protobuf#31768 ·
Maintainers usually reply within 1 day
-
Needs: Triage :mag: Platform: Android
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
react/react-native#60167 ·
Maintainers usually reply within 2 days
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 63/100
-
tests: ofx_render_test doesn't compile against FFmpeg 4.x (const AVCodec* to av_find_best_stream)Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NatronGitHub/openfx-io#221 · 1 comment ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
subsurface/subsurface#5145 ·
Maintainers usually reply within 2 days