brpc redis server 在单核情况下,耗时表现不佳

Open
#2,856 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
cpp, docker, kubernetes, redis

Research direction

No source file, test, or entry point is named. Start by reproducing the Redis-protocol request from 127.0.0.1 with the brpc server restricted to one CPU through Kubernetes request/limit settings, then compare P99 latency and scheduling behavior with an unrestricted server. Done means identifying the cause of the high single-core latency and documenting a validated solution.

Written by the indexing model from the issue text.

Description

背景描述

我们在线特征服务通过redis协议,请求一个同机部署的data_mesh服务。该data_mesh服务使用brpc搭建,redis协议。data_mesh收到同机的特征服务的请求后,进行协议转换然后请求远端存储。特征服务和 data_mesh 在同一个 POD,通过 k8s 进行部署。k8s 对 data_mesh 所在的 docker 进行了绑核,绑定到了 1 个 CPU 核心上。
绑核是通过 k8s 的 资源限制 request/limit 来做的,而非 set_cpu_affinitiy
特征服务请求 data_mesh 走的是 127.0.0.1, 不经过网卡

问题描述

我们发现特征服务请求 data_mesh 的耗时较高,哪怕是 data_mesh 不带任何业务(空跑一个 brpc server),同机调用的P99 latency 在 2ms 左右。 但 avg 耗时基本正常,比 ping 略高,在500us 左右。 我们的 qps 并不高,只有 50 左右。data_mesh 通过内置服务来看,资源使用情况也很空闲。
image

猜想

我们猜想,在 1 个核的情况下,brpc server 彻底失去了并行的能力,导致在 epoll thread 和 brpc worker 之间线程切换开销较大。

请问是否有更进一步的排查问题的方法以及解决方案?

Dominant language
C++
Stars
17.6k
Forks
4.2k
Avg merge
3d 17h
Merged PRs (30d)
72

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/brpc

All issues in apache/brpc

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.