[RDMA] 如何让请求 attachment 直接落入满足 NVMe PRP 要求的缓冲区?

Open
#3,730 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
cpp

Research direction

The issue points to RdmaEndpoint::PostRecv() and DoPostRecv() as the receive path, and to IOReserveAlignedBuf in #993 and #2942 as related work. Read those implementations and discussions first, then determine whether the current API supports application supplied DMA buffers or constrained receive segments. Done means identifying an existing approach or defining and validating a feasible design for PRP compatible payloads; the issue reports no end to end SPDK write test.

Written by the indexing model from the issue text.

Description

我们在评估用 bRPC 1.17 RDMA + SPDK NVMe driver 做存储服务,希望网卡收到的 payload 能直接提交给 NVMe,省掉从 attachment 到磁盘 I/O 缓冲区的 CPU 拷贝。

写入的磁盘 offset 和长度始终按 LBA 对齐,不涉及 read-modify-write。我们用 nvme id-ctrl -H 确认测试盘(Samsung PM9A3)报告 SGLS = 0,所以 NVMe 侧需要用 PRP 表示数据布局。

从源码看,RdmaEndpoint::PostRecv() 预先分配固定大小的 IOBuf 接收块,DoPostRecv() 每次提交一个接收 SGE。解析 RPC 后,attachment 可能从块内某个偏移开始,并跨多个接收块。即使这些块来自网卡和 NVMe 都能访问的 DMA 内存池,块间也可能在 NVMe 内存页中间跳到不连续的 DMA 地址;如果这个分段点也不在 LBA 边界,就无法在那里拆成两条写命令。

看到 #993 和 #2942 中的 IOReserveAlignedBuf 可以主动预留对齐空间,但没找到用它指定 incoming attachment 接收位置的接口。目前只核对了源码和控制器能力,还没有做端到端 SPDK 写盘验证。

想请教:

  1. bRPC 目前是否有办法把 RDMA attachment 直接接收到应用提供的 DMA buffer,或约束接收分段,让 payload 满足 PRP 布局?
  2. 如果没有,bRPC 是否适合提供一个可选模式,把 RPC 元数据与按页对齐的 payload 分开接收?还是建议由 bRPC 传控制信息,另用单边 RDMA 把 payload 放到应用预分配的 buffer?

如果已有相关实现或讨论,麻烦指个方向,谢谢!

Dominant language
C++
Stars
17.6k
Forks
4.2k
Avg merge
3d 17h
Merged PRs (30d)
72

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/brpc

All issues in apache/brpc

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.