Description
The BE process restarts unexpectedly. This issue tends to occur when starting the BE right after cluster deployment, and no such abnormal restarts will happen once it runs steadily.
I used the strace tool for analysis and found the signals captured by strace when the BE process restarted unexpectedly during the startup phase.
8439 rt_sigaction(SIGPIPE, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x7f922325f160}, NULL, 8) = 0 8439 tgkill(5071, 8439, SIGPIPE) = 0 8439 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_TKILL, si_pid=5071, si_uid=1000} --- 10023 +++ killed by SIGPIPE +++ 10022 +++ killed by SIGPIPE +++ 10021 +++ killed by SIGPIPE +++
Meanwhile, the specific thread execution information obtained by running the top command through the monitoring script is as follows:
HOST_TID=877055 CTID=8439 NAME=thrift_server STATE=S STACK=[[<0>] do_poll.constprop.0+0x302/0x3b0;[<0>] do_sys_poll+0x1dd/0x280;[<0>] __x64_sys_poll+0x3b/0x140;] HOST_TID=877055 CTID=8439 NAME=thrift_server STATE=S STACK=[[<0>] do_poll.constprop.0+0x302/0x3b0;[<0>] do_sys_poll+0x1dd/0x280;[<0>] __x64_sys_poll+0x3b/0x140;] HOST_TID=877055 CTID=8439 NAME=thrift_server STATE=S STACK=[[<0>] do_poll.constprop.0+0x302/0x3b0;[<0>] do_sys_poll+0x1dd/0x280;[<0>] __x64_sys_poll+0x3b/0x140;] HOST_TID=877055 CTID=8439 NAME=thrift_server STATE=S STACK=[[<0>] do_poll.constprop.0+0x302/0x3b0;[<0>] do_sys_poll+0x1dd/0x280;[<0>] __x64_sys_poll+0x3b/0x140;] HOST_TID=877055 CTID=8439 NAME=thrift_server STATE=S STACK=[[<0>] do_poll.constprop.0+0x302/0x3b0;[<0>] do_sys_poll+0x1dd/0x280;[<0>] __x64_sys_poll+0x3b/0x140;] HOST_TID=877055 CTID=8439 NAME=thrift_server STATE=S STACK=[[<0>] do_poll.constprop.0+0x302/0x3b0;[<0>] do_sys_poll+0x1dd/0x280;[<0>] __x64_sys_poll+0x3b/0x140;]
Analysis shows that the Thrift Server does not handle the SIGPIPE signal during startup, which causes the process to restart due to the default signal behavior. Is it feasible to add handling logic for the SIGPIPE signal in the source code?
be/src/service/daemon.cpp
` void init_signals() { auto ret = install_signal(SIGINT, sigterm_handler); if (ret < 0) { exit(-1); } ret = install_signal(SIGTERM, sigterm_handler); if (ret < 0) { exit(-1); }
//
signal(SIGPIPE, SIG_IGN);
} `
Steps to reproduce the behavior (Required)
Expected behavior (Required)
Real behavior (Required)
StarRocks version (Required)
StarRocks version 3.4.5