pingcap/tidb

ddl: give me the original trace stack when cancelling a ddl job

Open

#27,679 创建于 2021年8月30日

在 GitHub 查看
 (1 评论) (1 反应) (0 负责人)Go (6,186 fork)batch import
help wantedsig/sql-infratype/enhancement

仓库指标

Star
 (40,090 star)
PR 合并指标
 (平均合并 14天 4小时) (30 天内合并 346 个 PR)

描述

Enhancement

So far, when a ddl job encounter a error, TiDB will stores the error message into job and retry until the time limitation has been up to. When the limitation is reached, ddl owner will actively cancel this job and throw the error message out (not including the trace stack info)

Example

So what only you can see in the ddl owner's log is something like below:

[2021/08/30 17:22:17.713 +08:00] [INFO] [ddl_worker.go:701] 
["[ddl] DDL job is cancelled normally"] [worker="worker 1, tp general"] [error="pd unavailable"]

For the client side, what you can get is all about:

[2021/08/30 17:22:17.714 +08:00] [INFO] [conn.go:997] ["command dispatched failed"] [conn=3] [connInfo="id:3, addr:127.0.0.1:58552 status:10, collation:utf8mb4_unicode_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="    create placement policy x PRIMARY_REGION=\"cn-east-1\" REGIONS=\"cn-east-1,cn-east-2\"  LEADER_CONSTRAINTS=\"[+zone=cn-west-1]\"  CONSTRAINTS=\"[+disk=ssd]\""] [txn_mode=OPTIMISTIC] [err="[ddl:-1]pd unavailable\ngithub.com/pingcap/errors.AddStack
/home/arenatlx/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20210425183316-da1aaba5fb63/errors.go:174\ngithub.com/pingcap/errors.Trace
/home/arenatlx/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20210425183316-da1aaba5fb63/juju_adaptor.go:15\ngithub.com/pingcap/tidb/ddl.(*ddl).doDDLJob
/home/arenatlx/go/src/github.com/pingcap/tidb/ddl/ddl.go:610\ngithub.com/pingcap/tidb/ddl.(*ddl).CreatePlacementPolicy
/home/arenatlx/go/src/github.com/pingcap/tidb/ddl/ddl_api.go:6208\ngithub.com/pingcap/tidb/executor.(*DDLExec).executeCreatePlacementPolicy
/home/arenatlx/go/src/github.com/pingcap/tidb/executor/ddl.go:912\ngithub.com/pingcap/tidb/executor.(*DDLExec).Next
/home/arenatlx/go/src/github.com/pingcap/tidb/executor/ddl.go:237\ngithub.com/pingcap/tidb/executor.Next
/home/arenatlx/go/src/github.com/pingcap/tidb/executor/executor.go:286\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor
/home/arenatlx/go/src/github.com/pingcap/tidb/executor/adapter.go:581\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay
/home/arenatlx/go/src/github.com/pingcap/tidb/executor/adapter.go:462\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec
/home/arenatlx/go/src/github.com/pingcap/tidb/executor/adapter.go:411\ngithub.com/pingcap/tidb/session.runStmt
/home/arenatlx/go/src/github.com/pingcap/tidb/session/session.go:1674\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt
/home/arenatlx/go/src/github.com/pingcap/tidb/session/session.go:1568\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt
/home/arenatlx/go/src/github.com/pingcap/tidb/server/driver_tidb.go:219\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt
/home/arenatlx/go/src/github.com/pingcap/tidb/server/conn.go:1843\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery
/home/arenatlx/go/src/github.com/pingcap/tidb/server/conn.go:1707\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch
/home/arenatlx/go/src/github.com/pingcap/tidb/server/conn.go:1217\ngithub.com/pingcap/tidb/server.(*clientConn).Run
/home/arenatlx/go/src/github.com/pingcap/tidb/server/conn.go:979\ngithub.com/pingcap/tidb/server.(*Server).onConn
/home/arenatlx/go/src/github.com/pingcap/tidb/server/server.go:502\nruntime.goexit
/usr/lib/go/src/runtime/asm_amd64.s:1371"]

For the both side, what you have now is pd unavailable and rough front-side stack, but when and where the error is derived from? we need a detailed trace stack like front-side did in the ddl owner handling loop.

贡献者指南