Panic triggerred by the OOM-action output the same panic log with the panic that is unexpected · pingcap/tidb#27089

(5 comments) (0 reactions) (0 assignees)Go (6,186 forks)batch import

help wantedsig/executiontype/enhancement

Repository metrics

Stars: (40,090 stars)
PR merge metrics: (Avg merge 14d 4h) (346 merged PRs in 30d)

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

 select  count(*) FROM mysql.stats_fm_sketch WHERE table_id = 418;
+----------+
| count(*) |
+----------+
|   143268 |
+----------+
1 row in set (0.06 sec)

[adapter.go:335] ["execute sql panic"] [sql="delete from mysql.stats_fm_sketch where table_id = 418"] [stack="github.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:335\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:965\ngithub.com/pingcap/tidb/util/memory.(*PanicOnExceed).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/memory/action.go:128\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/store/copr.(*rateLimitAction).Action\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1111\ngithub.com/pingcap/tidb/util/memory.(*Tracker).Consume\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/memory/tracker.go:284\ngithub.com/pingcap/tidb/executor.(*DeleteExec).removeRow\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/delete.go:211\ngithub.com/pingcap/tidb/executor.(*DeleteExec).deleteOneRow\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/delete.go:62\ngithub.com/pingcap/tidb/executor.(*DeleteExec).deleteSingleTableByChunk\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/delete.go:112\ngithub.com/pingcap/tidb/executor.(*DeleteExec).Next\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/delete.go:50\ngithub.com/pingcap/tidb/executor.Next\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/executor.go:286\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:589\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:608\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:468\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:419\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/session/session.go:1635\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/session/session.go:1530\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/session/session.go:1250\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).DeleteTableStatsFromKV\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/gc.go:220\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).gcTableStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/gc.go:74\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).GCStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/gc.go:48\ngithub.com/pingcap/tidb/domain.(*Domain).updateStatsWorker\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/domain/domain.go:1312"]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

Release Version: v5.1.1
Edition: Community
Git Commit Hash: 797bddd25310ed42f0791c8eccb78be8cce2f502
Git Branch: heads/refs/tags/v5.1.1
UTC Build Time: 2021-07-28 11:00:26
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

Contributor guide

Research direction: Investigate the recursive calls in rateLimitAction.Action and the panic in PanicOnExceed.Action. Trace the code path in util/memory/action.go and store/copr/coprocessor.go to understand how the panic is triggered repeatedly.
Tech stack: go
Domain: backenddatabase
Issue type: Bug
Difficulty: 3
Estimated time: 1-3 hours
Activity status: Active
Clarity: Needs investigation
Prerequisites: GoTiDB internals
Newbie friendliness: 30