pingcap/tidb

need to handle "ER_WARN_ALLOWED_PACKET_OVERFLOWED" for some string functions

Open

#7,153 建立於 2018年7月25日

在 GitHub 查看
 (4 留言) (0 反應) (1 負責人)Go (40,090 star) (6,186 fork)batch import
component/expressionhelp wantedtype/compatibility

描述

Take this as an example:

select rpad('hello', 18446744073709551617, '1');

In MySQL, the result is NULL and a warning about "Result of rpad() was larger than max_allowed_packet" is returned:

MySQL(localhost:3306) > select rpad('hello', 18446744073709551617, '1');
+------------------------------------------+
| rpad('hello', 18446744073709551617, '1') |
+------------------------------------------+
| NULL                                     |
+------------------------------------------+
1 row in set, 3 warnings (0.01 sec)

MySQL(localhost:3306) > show warnings;
+---------+------+---------------------------------------------------------------------------+
| Level   | Code | Message                                                                   |
+---------+------+---------------------------------------------------------------------------+
| Warning | 1292 | Truncated incorrect DECIMAL value: '18446744073709551617'                 |
| Warning | 1292 | Truncated incorrect DECIMAL value: '18446744073709551617'                 |
| Warning | 1301 | Result of rpad() was larger than max_allowed_packet (4194304) - truncated |
+---------+------+---------------------------------------------------------------------------+
3 rows in set (0.00 sec)

MySQL(localhost:3306) > select version();
+-----------+
| version() |
+-----------+
| 5.7.22    |
+-----------+
1 row in set (0.00 sec)

In TiDB, this query results in a "lost connection" and the panic is "runtime error: makeslice: len out of range", and the panic stack is:

2018/07/25 16:33:10.027 conn.go:427: [error] lastCmd select rpad('hello', 18446744073709551617, '1'), runtime error: makeslice: len out of range, goroutine 743 [running]:
github.com/pingcap/tidb/server.(*clientConn).Run.func1(0xc43bbea000, 0xc4203bddff)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/server/conn.go:425 +0x10e
panic(0x1d54500, 0x2024180)
        /Users/jianzhang.zj/opt/goroot/src/runtime/panic.go:505 +0x229
strings.Repeat(0xc43c64253d, 0x1, 0x7ffffffffffffffb, 0xc4203bca18, 0x1)
        /Users/jianzhang.zj/opt/goroot/src/strings/strings.go:543 +0x83
github.com/pingcap/tidb/expression.(*builtinRpadSig).evalString(0xc422664f40, 0x0, 0x0, 0x0, 0x17f2c44, 0xc422664f40, 0x205eb00, 0xc43e274000)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/expression/builtin_string.go:1959 +0x38c
github.com/pingcap/tidb/expression.(*ScalarFunction).EvalString(0xc43d1e0aa0, 0x205eb00, 0xc43e274000, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4203bcca8, 0x1741c17)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/expression/scalar_function.go:216 +0x4c
github.com/pingcap/tidb/expression.(*ScalarFunction).Eval(0xc43d1e0aa0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/expression/scalar_function.go:188 +0x481
github.com/pingcap/tidb/expression.foldConstant(0x205df00, 0xc43d1e0aa0, 0x1e18b60, 0xc43c642501, 0xc43d1e0aa0)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/expression/constant_fold.go:102 +0x5e2
github.com/pingcap/tidb/expression.FoldConstant(0x205df00, 0xc43d1e0aa0, 0xc43c642518, 0x4)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/expression/constant_fold.go:33 +0x35
github.com/pingcap/tidb/expression.NewFunction(0x205eb00, 0xc43e274000, 0xc43c642518, 0x4, 0xc43c6406c0, 0xc422664ec0, 0x3, 0x4, 0x1e18aa0, 0xc422664ec0, ...)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/expression/scalar_function.go:98 +0x341
github.com/pingcap/tidb/plan.(*expressionRewriter).funcCallToExpression(0xc421a80d00, 0xc43c8e87e0)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/expression_rewriter.go:1198 +0x174
github.com/pingcap/tidb/plan.(*expressionRewriter).Leave(0xc421a80d00, 0x20313e0, 0xc43c8e87e0, 0x2048e40, 0xc438357e40, 0x1aa0b01)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/expression_rewriter.go:746 +0x86a
github.com/pingcap/tidb/ast.(*FuncCallExpr).Accept(0xc43c8e87e0, 0x202cae0, 0xc421a80d00, 0x1a2fd5c, 0x205e800, 0xc421a80c80)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/ast/functions.go:363 +0x1bf
github.com/pingcap/tidb/plan.(*planBuilder).rewriteExprNode(0xc43b7e3e00, 0xc421a80d00, 0x2048740, 0xc43c8e87e0, 0xc4203bd301, 0x1010559, 0xc422664e40, 0x40, 0x38, 0x1dedf60, ...)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/expression_rewriter.go:132 +0x54
github.com/pingcap/tidb/plan.(*planBuilder).rewriteWithPreprocess(0xc43b7e3e00, 0x2048740, 0xc43c8e87e0, 0x205e800, 0xc421a80c80, 0x0, 0x1eba401, 0x0, 0x0, 0x0, ...)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/expression_rewriter.go:104 +0x12b
github.com/pingcap/tidb/plan.(*planBuilder).rewrite(0xc43b7e3e00, 0x2048740, 0xc43c8e87e0, 0x205e800, 0xc421a80c80, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/expression_rewriter.go:80 +0x96
github.com/pingcap/tidb/plan.(*planBuilder).buildProjection(0xc43b7e3e00, 0x205e800, 0xc421a80c80, 0xc42402e6e0, 0x1, 0x1, 0x0, 0x1, 0x1, 0x1e17360)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/logical_plan_builder.go:572 +0x220
github.com/pingcap/tidb/plan.(*planBuilder).buildSelect(0xc43b7e3e00, 0xc4210bb4a0, 0x0, 0x0)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/logical_plan_builder.go:1664 +0x309
github.com/pingcap/tidb/plan.(*planBuilder).build(0xc43b7e3e00, 0x2031ca0, 0xc4210bb4a0, 0xc421a293b0, 0x205eb00)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/planbuilder.go:155 +0x603
github.com/pingcap/tidb/plan.Optimize(0x205eb00, 0xc43e274000, 0x2031ca0, 0xc4210bb4a0, 0x2048fc0, 0xc421a293b0, 0x2a7bd00, 0x0, 0x0, 0x2f)
        /Users/jianzhang.zj/opt/gopath/src/github.com/pingcap/tidb/plan/optimize
2018/07/25 16:33:10.027 server.go:316: [info] con:1 close connection

What we need to do is not allocate a buffer larger than "max_allowed_packet" for these string functions:

  • to_base64 @supernan1994 #7266
  • from_base64 @supernan1994 #7409
  • concat @amyangfei #11137
  • concat_ws @amyangfei #11137
  • replace
  • insert @eurekaka #7502
  • repeat @hhxcc #7181
  • space @hhxcc #7210
  • rpad @zz-jason #7171
  • lpad @zz-jason #7171
  • cast
  • export_set
  • quote

All the above functions implemented in MySQL are located in the "sql/item_strfunc.cc" file, you can grep the function name to see specific implementations.

All the above functions can be found in TiDB from the source file "expression/builtin.go".

貢獻者指南

技術棧
gosql
領域
database
議題類型
bug
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-2 days
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
Go programmingMySQL query warning handlingTiDB expression framework
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
60
研究方向
To fix this issue, start by examining the implementations of `replace`, `cast`, `export set`, and `quote` in the file `expression/builtin string.go`. These functions should check the result length against `max allowed packet` and return NULL with a warning if exceeded, similar to how `rpad` was fixed in PR #7171. Refer to the MySQL source file `sql/item strfunc.cc` for the expected behavior. The key is to avoid allocating a buffer larger than `max allowed packet` to prevent the 'makeslice: len out of range' panic.