tikv/tikv

TiKV spend more than 20 minutes to (re)start

Offen

#10.444 geöffnet am 23.06.2021

 (7 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)Rust (2.189 Forks)batch import
help wantedseverity/moderatesig/rafttype/bug

Repository-Metriken

Stars
 (15.922 Sterne)
PR-Merge-Metriken
 (Durchschn. Merge 28T 9h) (68 gemergte PRs in 30 T)

Beschreibung

Bug Report

What version of TiKV are you using?

2ab5b40b2bfdceaeaf61806ffe5e24b57d5c3ebb

[root@7b6cbaa65e59 log]# ../bin/tikv-server -V
TiKV
Release Version:   5.1.0
Edition:           Community
Git Commit Hash:   2ab5b40b2bfdceaeaf61806ffe5e24b57d5c3ebb
Git Commit Branch: heads/refs/tags/v5.1.0
UTC Build Time:    2021-06-22 12:17:22
Rust Version:      rustc 1.53.0-nightly (16bf626a3 2021-04-14)
Enable Features:   jemalloc mem-profiling portable sse protobuf-codec test-engines-rocksdb cloud-aws cloud-gcp
Profile:           dist_release

What operating system and CPU are you using?

Steps to reproduce

I'm not sure if the following steps can reproduce the problem.

  1. Run write-stress testcase. /tmp/hhhhh/bin/write-stress -tidb-server xxx -case-name append -tables 128 -pad-length 4000000 -run-time 3h -concurrency 100
  2. The TiKV restarts because of OOM.
  3. The problem may occur.

What did you expect?

TiKV restart and serve immediatelly.

What did happened?

The log shows that the store spends 1287.795972388s to do initialization.

[2021/06/23 17:48:48.309 +08:00] [INFO] [raft.rs:1088] ["became follower at term 5"] [term=5] [raft_id=12471] [region_id=12468]
[2021/06/23 17:48:48.309 +08:00] [INFO] [raft.rs:389] [newRaft] [peers="Configuration { incoming: Configuration { voters: {12469, 12470, 12471} }, outgoing: Configuration { voters: {} } }"] ["last term"=5] ["last index"=5] [applied=5] [commit=5] [term=5] [raft_id=12471] [region_id=12468]
[2021/06/23 17:48:48.309 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 12471."] [id=12471] [raft_id=12471] [region_id=12468]
[2021/06/23 17:58:41.793 +08:00] [INFO] [util.rs:536] ["connecting to PD endpoint"] [endpoints=http://10.0.2.102:2379]
[2021/06/23 18:08:41.795 +08:00] [INFO] [util.rs:536] ["connecting to PD endpoint"] [endpoints=http://10.0.2.102:2379]
[2021/06/23 18:10:14.932 +08:00] [INFO] [store.rs:1030] ["start store"] [takes=1287.795972388s] [merge_count=5] [applying_count=0] [tombstone_count=579] [region_count=3018] [store_id=4]
[2021/06/23 18:10:14.936 +08:00] [INFO] [store.rs:1078] ["cleans up garbage data"] [takes=3.025005ms] [garbage_range_count=2440] [store_id=4]
[2021/06/23 18:10:14.993 +08:00] [INFO] [future.rs:146] ["starting working thread"] [worker=pd-worker]
[2021/06/23 18:10:14.994 +08:00] [WARN] [store.rs:1376] ["set thread priority for raftstore failed"] [error="Os { code: 13, kind: PermissionDenied, message: \"Permission denied\" }"]

image

Contributor Guide