brpc redis server 在单核情况下,耗时表现不佳
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- cpp, docker, kubernetes, redis
- Domain
- backend, performance
Research direction
No source file, test, or entry point is named. Start by reproducing the Redis-protocol request from 127.0.0.1 with the brpc server restricted to one CPU through Kubernetes request/limit settings, then compare P99 latency and scheduling behavior with an unrestricted server. Done means identifying the cause of the high single-core latency and documenting a validated solution.
Written by the indexing model from the issue text.
Description
背景描述
我们在线特征服务通过redis协议,请求一个同机部署的data_mesh服务。该data_mesh服务使用brpc搭建,redis协议。data_mesh收到同机的特征服务的请求后,进行协议转换然后请求远端存储。特征服务和 data_mesh 在同一个 POD,通过 k8s 进行部署。k8s 对 data_mesh 所在的 docker 进行了绑核,绑定到了 1 个 CPU 核心上。
绑核是通过 k8s 的 资源限制 request/limit 来做的,而非 set_cpu_affinitiy
特征服务请求 data_mesh 走的是 127.0.0.1, 不经过网卡
问题描述
我们发现特征服务请求 data_mesh 的耗时较高,哪怕是 data_mesh 不带任何业务(空跑一个 brpc server),同机调用的P99 latency 在 2ms 左右。 但 avg 耗时基本正常,比 ping 略高,在500us 左右。 我们的 qps 并不高,只有 50 左右。data_mesh 通过内置服务来看,资源使用情况也很空闲。
猜想
我们猜想,在 1 个核的情况下,brpc server 彻底失去了并行的能力,导致在 epoll thread 和 brpc worker 之间线程切换开销较大。
请问是否有更进一步的排查问题的方法以及解决方案?
- Dominant language
- C++
- Stars
- 17.6k
- Forks
- 4.2k
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 72
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/brpc
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 2 days
-
endpoint2hostname() prints a byte-swapped port for IPv6 endpointsPossibly taken @0z5a claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 2 days
-
perf(butil): ARM64 (aarch64) performance and modernisation improvements across atomics, CRC32C, spin-wait hints, and fast timersPossibly taken @jimmystewpot claimed this 2 days ago. Open
Difficulty 5/5 Over a week Newbie friendliness 12/100
Maintainers usually reply within 2 days
-
file:// naming service takes a comment after the address as the tagPossibly taken @0z5a claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
Maintainers usually reply within 2 days
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Maintainers usually reply within 2 days
Similar issues
-
agent:Windows bug in progress pri:medium sev:low testing windows
Difficulty 1/5 Under an hour Newbie friendliness 91/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
protocolbuffers/protobuf#31768 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
godotengine/godot#125651 ·
Maintainers usually reply within 1 day
-
Android: with view recycling on, a recycled View keeps its nextFocusLeftPossibly taken A pull request linked to this issue is open or already merged. OpenNeeds: Triage :mag: Platform: Android
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
react/react-native#60167 ·
Maintainers usually reply within 2 days
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 63/100