[client-v2] Enum label parsing: backslash escapes in a type name are decoded by dropping the backslash, so labels with a newline, tab or NUL are wrong
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
調査の方向性
Start in ClickHouseUtils.unescape (clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseUtils.java, around line 511), which ClickHouseEnum uses to decode Enum8/Enum16 labels from type strings. Replace the drop-the-backslash rule with the server's escape mapping, keeping unknown escapes intact and not consuming digits after \0. Run the existing ClickHouseUtilsTest and ClickHouseColumnTest, plus the issue's parser examples such as Enum8('a\nb' = 1) expecting label a LF b. Done when those examples decode correctly and the existing tests still pass.
索引モデルが issue の本文から書いたものです。
説明
Describe the bug
ClickHouseUtils.unescape (clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseUtils.java:511-540) is used by ClickHouseEnum to decode Enum8 / Enum16 labels out of a type string (ClickHouseEnum.java:46). It does not implement the server's string-literal escape rules: for any backslash-X sequence it simply drops the backslash and keeps X.
The server writes control characters in a type name as C-style escapes. SELECT toTypeName(CAST(1 AS Enum8('a\nb' = 1))) returns the type string Enum8('a\nb' = 1) where \n is backslash + n, and the real label is a LF b. The client decodes that to the 3-character string anb.
So for every enum label that contains a control character, the client silently returns a label that is different from the one the server stores, and an insert with the correct label throws because the label is not in the map.
What is right and what is wrong today, for the type strings the server itself emits:
| Type string from the server | Server label (hex) | Client label (hex) | |
|---|---|---|---|
Enum8('a\nb' = 1) |
610A62 |
616E62 (anb) |
wrong |
Enum8('a\tb' = 1) |
610962 |
617462 (atb) |
wrong |
Enum8('a\0b' = 1) |
610062 |
613062 (a0b) |
wrong |
Enum8('a\\zb' = 1) |
615C7A62 (a\zb) |
615C7A62 |
ok |
Enum8('it\'s' = 1) |
69742773 (it's) |
69742773 |
ok |
Only the doubled backslash and the escaped quote happen to come out right, because for those two the server rule and "drop the backslash" agree. \a, \b, \e, \f, \n, \r, \t, \v, \0, \xHH are all decoded wrong, and an escape the server keeps verbatim loses its backslash.
The same decoder is on both sides of the round trip, so getEnumConstants().value("a\nb") (the insert path, SerializerUtils.java:865) also fails for a label the server really has.
This is the Java counterpart of ClickHouse/clickhouse-cs#729 (that client uses .NET Regex.Unescape, a different wrong rule with a partly different symptom set).
ClickHouse server version
26.10.3.7, verified against a running server.
Reproduction
End-to-end, client-v2 over HTTP (TestNG, client-v2/src/test/java/...):
package com.clickhouse.client;
import com.clickhouse.client.api.Client;
import com.clickhouse.client.api.query.GenericRecord;
import org.testng.Assert;
import org.testng.annotations.Test;
import java.util.List;
public class EnumEscapeTest {
private static String hex(String s) {
StringBuilder sb = new StringBuilder();
for (char c : s.toCharArray()) {
sb.append(String.format("%02X", (int) c));
}
return sb.toString();
}
@Test(groups = { "unit" })
public void testEnumLabelWithNewline() throws Exception {
try (Client client = new Client.Builder()
.addEndpoint("http://localhost:8123")
.setUsername("default").setPassword("")
.compressServerResponse(false)
.build()) {
client.execute("DROP TABLE IF EXISTS scratch_enum_nl").get();
client.execute("CREATE TABLE scratch_enum_nl (e Enum8('a\\nb' = 1)) ENGINE = Memory").get();
client.execute("INSERT INTO scratch_enum_nl VALUES (1)").get();
List<GenericRecord> serverSide = client.queryAll("SELECT hex(toString(e)) AS h FROM scratch_enum_nl");
String serverHex = serverSide.get(0).getString("h");
List<GenericRecord> clientSide = client.queryAll("SELECT e FROM scratch_enum_nl");
String clientLabel = clientSide.get(0).getString("e");
Assert.assertEquals(hex(clientLabel), serverHex);
}
}
}
Result:
java.lang.AssertionError: expected [610A62] but found [616E62]
Expected 610A62 (a, LF, b) — what the server stores. Actual 616E62 (anb).
Parser-level, no server needed (clickhouse-data, these are the exact type strings toTypeName() returns):
ClickHouseColumn.of("e", "Enum8('a\\nb' = 1)").getEnumConstants().name(1); // "anb", expected "a\nb"
ClickHouseColumn.of("e", "Enum8('a\\tb' = 1)").getEnumConstants().name(1); // "atb", expected "a\tb"
ClickHouseColumn.of("e", "Enum8('a\\0b' = 1)").getEnumConstants().name(1); // "a0b", expected "a\0b"
ClickHouseColumn.of("e", "Enum8('a\\\\zb' = 1)").getEnumConstants().name(1); // "a\\zb", ok
ClickHouseColumn.of("e", "Enum8('it\\'s' = 1)").getEnumConstants().name(1); // "it's", ok
Suggested fix
In ClickHouseUtils.unescape (clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseUtils.java:511), decode with the server's rules instead of unconditionally dropping the backslash:
- a doubled quote is one quote (already correct);
\a,\b,\e,\f,\n,\r,\t,\v,\0, doubled backslash, escaped quotes and backtick, and\xHHmap to their byte values;- any other escape keeps the backslash (
\zstays\z); \0never consumes the digits that follow it (\012is NUL then12).
unescape is also reached from ClickHouseUtils.readValueArray (ClickHouseUtils.java:1449), so any change there should be checked against the existing ClickHouseUtilsTest / ClickHouseColumnTest cases.
Link
Counterpart issue in the .NET client: https://github.com/ClickHouse/clickhouse-cs/issues/729
- 主要言語
- Java
- スター
- 1.6k
- フォーク
- 649
- 平均マージ
- 1日 20時間
- マージ済み PR(30日)
- 39
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
ClickHouse/clickhouse-java のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
ClickHouse/clickhouse-java#3349 ·
メンテナーはふだん 1 日以内に返信
-
bug jdbc-v2
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
ClickHouse/clickhouse-java#3347 ·
メンテナーはふだん 1 日以内に返信
-
bug jdbc-v2
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
ClickHouse/clickhouse-java#3326 ·
メンテナーはふだん 1 日以内に返信
-
[client-v2] Enum8/Enum16 insert accepts a non-member integer and uses Enum.ordinal(), silently storing values that are not enum members対応中かも @jaya232 が 6 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
ClickHouse/clickhouse-java#3304 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
bug jdbc-v2
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
ClickHouse/clickhouse-java#3214 ·
メンテナーはふだん 1 日以内に返信
ClickHouse/clickhouse-java の issue をすべて見る
似ている issue
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
bug:unconfirmed
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
UniversalMediaServer/UniversalMediaServer#6473 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
StarRocks/starrocks#83246 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
メンテナーはふだん 2 日以内に返信