tikv/tikv

TiKV spend more than 20 minutes to (re)start

Aperta

#10.444 aperta il 23 giu 2021

 (7 commenti) (0 reazioni) (0 assegnatari)Rust (2189 fork)batch import
help wantedseverity/moderatesig/rafttype/bug

Metriche repository

Star
 (15.922 stelle)
Metriche merge PR
 (Merge medio 28g 9h) (68 PR mergiate in 30 g)

Descrizione

Bug Report

What version of TiKV are you using?

2ab5b40b2bfdceaeaf61806ffe5e24b57d5c3ebb

[root@7b6cbaa65e59 log]# ../bin/tikv-server -V
TiKV
Release Version:   5.1.0
Edition:           Community
Git Commit Hash:   2ab5b40b2bfdceaeaf61806ffe5e24b57d5c3ebb
Git Commit Branch: heads/refs/tags/v5.1.0
UTC Build Time:    2021-06-22 12:17:22
Rust Version:      rustc 1.53.0-nightly (16bf626a3 2021-04-14)
Enable Features:   jemalloc mem-profiling portable sse protobuf-codec test-engines-rocksdb cloud-aws cloud-gcp
Profile:           dist_release

What operating system and CPU are you using?

Steps to reproduce

I'm not sure if the following steps can reproduce the problem.

  1. Run write-stress testcase. /tmp/hhhhh/bin/write-stress -tidb-server xxx -case-name append -tables 128 -pad-length 4000000 -run-time 3h -concurrency 100
  2. The TiKV restarts because of OOM.
  3. The problem may occur.

What did you expect?

TiKV restart and serve immediatelly.

What did happened?

The log shows that the store spends 1287.795972388s to do initialization.

[2021/06/23 17:48:48.309 +08:00] [INFO] [raft.rs:1088] ["became follower at term 5"] [term=5] [raft_id=12471] [region_id=12468]
[2021/06/23 17:48:48.309 +08:00] [INFO] [raft.rs:389] [newRaft] [peers="Configuration { incoming: Configuration { voters: {12469, 12470, 12471} }, outgoing: Configuration { voters: {} } }"] ["last term"=5] ["last index"=5] [applied=5] [commit=5] [term=5] [raft_id=12471] [region_id=12468]
[2021/06/23 17:48:48.309 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 12471."] [id=12471] [raft_id=12471] [region_id=12468]
[2021/06/23 17:58:41.793 +08:00] [INFO] [util.rs:536] ["connecting to PD endpoint"] [endpoints=http://10.0.2.102:2379]
[2021/06/23 18:08:41.795 +08:00] [INFO] [util.rs:536] ["connecting to PD endpoint"] [endpoints=http://10.0.2.102:2379]
[2021/06/23 18:10:14.932 +08:00] [INFO] [store.rs:1030] ["start store"] [takes=1287.795972388s] [merge_count=5] [applying_count=0] [tombstone_count=579] [region_count=3018] [store_id=4]
[2021/06/23 18:10:14.936 +08:00] [INFO] [store.rs:1078] ["cleans up garbage data"] [takes=3.025005ms] [garbage_range_count=2440] [store_id=4]
[2021/06/23 18:10:14.993 +08:00] [INFO] [future.rs:146] ["starting working thread"] [worker=pd-worker]
[2021/06/23 18:10:14.994 +08:00] [WARN] [store.rs:1376] ["set thread priority for raftstore failed"] [error="Os { code: 13, kind: PermissionDenied, message: \"Permission denied\" }"]

image

Guida contributor