[client-v2] ACCESS_DENIED (HTTP 403) error text is lost with the default response compression: "<Unreadable error message> (transport error: 403)"

Start with HttpAPIClientHelper.wrapResponseEntity near line 1121 and trace how readClickHouseError and readNotClickHouseError handle compressed and plain text bodies. Check CompressedBlockInputStream's handling of short bodies, then add coverage for compressed ACCESS_DENIED responses and short plain-text 401/403 bodies, with and without X-ClickHouse-Exception-Code. Done means the server error text remains readable for both cases; the issue gives no test file path.

Description

With the default settings (compress=1, response compression on), client-v2 does not read the server error text when the server returns HTTP 403 for a query-level error, for example ACCESS_DENIED (497). The error code comes from the X-ClickHouse-Exception-Code header, but the message is lost:

• ClickHouse 26.9+ (default response codec ZSTD): Code: 497. DB::Exception: <Unreadable error message> (transport error: 403)
• ClickHouse 26.8 and older (default codec LZ4): the message is readable but contains binary garbage, for example ... grant SHOW COLUMNS ON default.s\0�+t. instead of default.lead_t.

jdbc-v2 is affected with its default URL (for example on INSERT, SELECT, and the RowBinary writer's DESCRIBE TABLE).

With compressServerResponse(false) (jdbc compress=0) or useHttpCompression(true), the message is correct. Errors with HTTP 400/500 are correct with compression on. Authentication failures (516, also HTTP 403) are correct too.

Found by automated analysis of the client while working on #3326. Verified against live servers 26.8.1.2041, 26.9.4.3 and 26.10.3.7 (not by inspection only).

Steps to reproduce

1. CREATE USER lead_noacc IDENTIFIED WITH plaintext_password BY 'pw' (no grants), CREATE TABLE default.lead_t (x Int32) ENGINE = MergeTree ORDER BY x.
2. As lead_noacc, with the default client settings, run DESCRIBE TABLE default.lead_t or SELECT * FROM default.lead_t.
3. Look at the exception message.

Error Log or Exception StackTrace

Output of the code example below, main f9586aba0, ClickHouse 26.9.4.3:

[v2 default        ] DESCRIBE TABLE default.lead_t -> ServerException: Code: 497. DB::Exception: <Unreadable error message> (transport error: 403) (queryId= f54e9de5-...)
[v2 compress=false ] DESCRIBE TABLE default.lead_t -> ServerException: Code: 497. DB::Exception: lead_noacc: Not enough privileges. To execute this query, it's necessary to have the grant SHOW COLUMNS ON default.lead_t. (ACCESS_DENIED) (version 26.9.4.3 (official build))
[v2 httpCompression] DESCRIBE TABLE default.lead_t -> ServerException: Code: 497. DB::Exception: lead_noacc: Not enough privileges. ... (ACCESS_DENIED) ...
[v2 default        ] SELECT * FROM default.lead_t -> ServerException: Code: 497. DB::Exception: <Unreadable error message> (transport error: 403) (queryId= f2e94e32-...)
[v2 default        ] SELECT throwIf(1, 'boom-400') -> ServerException: Code: 395. DB::Exception: boom-400: while executing ... (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) ...
[v2 default wrong-pw] SELECT 1 -> ServerException: Code: 516. DB::Exception: lead_noacc: Authentication failed: password is incorrect, or there is no user with such name. (AUTHENTICATION_FAILED)
[jdbc default  ] INSERT -> SQLException: Code: 497. DB::Exception: <Unreadable error message> (transport error: 403) (queryId= 393ab454-...)
[jdbc compress=0] INSERT -> SQLException: Code: 497. DB::Exception: lead_noacc: Not enough privileges. To execute this query, it's necessary to have the grant INSERT ON default.lead_t. (ACCESS_DENIED) ...

Same program on ClickHouse 26.8.1.2041 (LZ4), default settings:

[v2 default        ] DESCRIBE TABLE default.lead_t -> ServerException: Code: 497. DB::Exception: lead_noacc: Not enough privileges. To execute this query, it's necessary to have the grant SHOW COLUMNS ON default.s^@M-oM-?M-=+t. (ACCESS_DENIED) ...
[jdbc default  ] INSERT -> SQLException: Code: 497. DB::Exception: lead_noacc: Not enough privileges. To execute this query, it's necessary to have the grant INSERT ON default.m^@M-oM-?M-=+t. (ACCESS_DENIED) ...

(cat -v output: the table name is replaced by raw LZ4 bytes.)

Expected Behaviour

The exception message contains the server text, as it does with compress=0. The server sends it: with compress=1 the 403 body for ACCESS_DENIED is a compressed block (method byte at offset 16 is 0x90 = ZSTD on 26.9), and the client must decompress it like any other compressed response:

$ curl -s -D - -u lead_noacc:pw 'http://clickhouse:8123/?compress=1&query=DESCRIBE+TABLE+default.lead_t' -o body
HTTP/1.1 403 Forbidden
X-ClickHouse-Exception-Code: 497
$ od -A d -t x1 -N 24 body
0000000 e8 28 96 52 39 a5 8a 6b 98 c2 07 ec b1 2c a5 a3
0000016 90 bd 00 00 00 c9 00 00 00 28 b5 2f

An authentication failure is also HTTP 403, but its body is plain text, because the server rejects the request before the compressed output exists:

$ curl -s -D - -u lead_noacc:bad 'http://clickhouse:8123/?compress=1&query=SELECT+1' -o body
HTTP/1.1 403 Forbidden
X-ClickHouse-Exception-Code: 516
$ od -A d -c -N 16 body
0000000   C   o   d   e   :       5   1   6   .       D   B   :   :   E

So the HTTP status does not tell if the body is compressed.

Root cause. HttpAPIClientHelper.wrapResponseEntity (client-v2/src/main/java/com/clickhouse/client/api/internal/HttpAPIClientHelper.java:1121) does not wrap the response in CompressedBlockEntity when the status is 401 or 403:

if (serverCompression && !(httpStatus == HttpStatus.SC_FORBIDDEN || httpStatus == HttpStatus.SC_UNAUTHORIZED)) {

readClickHouseError (:488) then scans the raw compressed bytes for 497. DB::Exception:. With ZSTD the text is entropy-coded, the prefix is not found, and :543-544 puts <Unreadable error message>. With LZ4 the literals are mostly plain text, so the prefix is found and the LZ4 match tokens go into the message.

Suggested fix. Select decompression by the body content, not by the HTTP status. CompressedBlockInputStream already detects LZ4 / ZSTD / NONE and throws ClientException("Invalid LZ4 magic byte ...") for a plain-text body (CompressedBlockInputStream.java:113-120). readClickHouseError and readNotClickHouseError already catch it and read the body raw from getHeaderBuffer(). This is how plain-text 500 errors are read today with compression on. So removing the 401/403 condition is sufficient for the normal cases. I checked this with a local patch (the condition removed): on 26.8, 26.9 and 26.10 the ACCESS_DENIED message is correct, and the AUTHENTICATION_FAILED (516, plain-text 403) message stays correct.

Contrast case that must stay readable: a plain-text 401/403 body shorter than the 25-byte block header (for example from a proxy). readFully (CompressedBlockInputStream.java:89) throws IOException("Incomplete read ...") for it, which the error readers do not handle as plain text. A test for a short plain-text 401/403 body, with and without X-ClickHouse-Exception-Code, should cover this. (#3322 changes the partial-header handling in readNotClickHouseError, but not the 401/403 condition in wrapResponseEntity.)

Code Example

// as default: CREATE USER lead_noacc IDENTIFIED WITH plaintext_password BY 'pw'
//             CREATE TABLE default.lead_t (x Int32) ENGINE = MergeTree ORDER BY x
try (Client c = new Client.Builder().addEndpoint("http://localhost:8123")
        .setUsername("lead_noacc").setPassword("pw")
        .build()) {                                   // compressServerResponse(true) is the default
    c.query("DESCRIBE TABLE default.lead_t").get().close();
} catch (Exception e) {
    System.out.println(e.getMessage());
    // actual (26.9+):  Code: 497. DB::Exception: <Unreadable error message> (transport error: 403) ...
    // expected:        Code: 497. DB::Exception: lead_noacc: Not enough privileges. ... (ACCESS_DENIED) ...
}

// jdbc-v2, default URL
Properties p = new Properties();
p.setProperty("user", "lead_noacc");
p.setProperty("password", "pw");
try (Connection conn = new com.clickhouse.jdbc.Driver().connect("jdbc:clickhouse://localhost:8123/default", p);
     Statement st = conn.createStatement()) {
    st.execute("INSERT INTO default.lead_t VALUES (1)");
} catch (SQLException e) {
    System.out.println(e.getMessage()); // same <Unreadable error message> (transport error: 403)
}

Configuration

Client Configuration

// defaults: compressServerResponse(true), useHttpCompression(false)
// jdbc-v2: default URL (compress not set)

Environment

• Cloud
• Client version: client-v2 / jdbc-v2 0.12.0-rc1-SNAPSHOT (main f9586aba0)
• Language version: Java 17
• OS: Linux (Docker)

ClickHouse Server

• ClickHouse Server version: 26.9.4.3 and 26.10.3.7 (unreadable), 26.8.1.2041 (message corrupted)
• ClickHouse Server non-default settings, if any: none
• CREATE TABLE statements for tables involved: CREATE TABLE default.lead_t (x Int32) ENGINE = MergeTree ORDER BY x
• Sample data for all these tables: not needed (the user has no grants)