MPP query fails with "Cannot LZ4_decompress_safe" when one probe row matches many build rows

Aperta
#11,167 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, databases

Direzione di ricerca

Start with JoinPartition.cpp at Adder<KIND, Strictness::All, Map>::addFound, then trace how HashPartitionWriter flushes partitions and how CompressionCodecLZ4::doCompressData handles oversized input. Reproduce with the SQL query in the report using legacy hash join and FAST exchange compression. Done means the query returns the expected count and maximum value without decompression or packet-size errors.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

component/compute

Bug Report

1. Minimal reproduce step (Required)

Use a cluster with 3 TiFlash nodes.

CREATE TABLE a (k BIGINT NOT NULL, c BIGINT NOT NULL);
CREATE TABLE b (k BIGINT NOT NULL, s VARCHAR(64) NOT NULL);
CREATE TABLE c (c BIGINT NOT NULL);
INSERT INTO a VALUES (1, 3);
INSERT INTO c VALUES (3);
SET SESSION cte_max_recursion_depth = 10000000;
INSERT INTO b WITH RECURSIVE g(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 1000000)
SELECT 1, LPAD(n, 30, 'x') FROM g;
SET SESSION tidb_dml_type = 'bulk';
INSERT INTO b SELECT * FROM b; -- Do this 6 times. Then b has 64,000,000 rows.
ALTER TABLE a SET TIFLASH REPLICA 1;
ALTER TABLE b SET TIFLASH REPLICA 1;
ALTER TABLE c SET TIFLASH REPLICA 1;
-- Wait until the TiFlash replicas are available.
ANALYZE TABLE a, b, c;

SET SESSION tidb_isolation_read_engines = 'tiflash,tidb';
SET SESSION tidb_enforce_mpp = ON;
SET SESSION tidb_broadcast_join_threshold_count = 0;
SET SESSION tidb_broadcast_join_threshold_size = 0;
SET SESSION tiflash_hash_join_version = 'legacy';
SET SESSION mpp_exchange_compression_mode = 'FAST';
SELECT /*+ HASH_JOIN_BUILD(b) */ STRAIGHT_JOIN count(*), max(b.s)
FROM a JOIN b ON a.k = b.k JOIN c ON a.c = c.c;

The first join builds on b and probes the one row of a. An ExchangeSender sends the join output to the second
join by a hash partition on a.c. If the error is Packet is too large to send, the partition stays on the same
node. Then use another value for a.c and c.c, for example 2.

2. What did you expect to see? (Required)

The query returns 64000000 and the largest value of b.s.

3. What did you see instead (Required)

The query fails after 11 s:

ERROR 1105 (HY000): other error for mpp stream: Code: 271, e.displayText() = DB::Exception: Cannot LZ4_decompress_safe: ... ExchangeReceiver_42
Setting Result
FAST, the partition goes to another node Fails: Cannot LZ4_decompress_safe (Code 271)
NONE, or the partition stays on the same node Fails: Packet is too large to send, size : 3520000195 (Code 49)
tiflash_hash_join_version = 'optimized' Succeeds in 6–8 s

The packet has 64,000,000 rows × 55 bytes + 195 bytes. Thus, all output rows of the join go into one packet.

The same failure occurs in JCC-H q7 at scale factor 1,000 (TiFlash mode). The join lineitem JOIN orders builds on
lineitem, and one row of orders matches 75,200,000 rows. The packets have 4,211,323,072 and 8,874,106,922 bytes.

Cause:

  1. Adder<KIND, Strictness::All, Map>::addFound (JoinPartition.cpp) checks max_block_size only when the output
    block already has rows. Thus, the first probe row of a block adds all its matches to the block.
  2. HashPartitionWriter uses max_buffered_rows and max_buffered_bytes only to start a flush. A flush writes all
    buffered rows of a partition as one packet. All rows of this block have the same partition key.
  3. CompressionCodecLZ4::doCompressData does not check the result of LZ4_compress_fast. For an input larger than
    LZ4_MAX_INPUT_SIZE (2,113,929,216 bytes), LZ4 returns 0, and the receiver fails in LZ4_decompress_safe.
    CompressionCodecLZ4HC checks this result. Without compression, checkPacketSize (limit 2^31 bytes) gives the
    real cause.

Hash join v2 stops each output block at max_block_size, also inside the matches of one probe row. But the default
of tiflash_hash_join_version is legacy.

4. What is your TiFlash version? (Required)

v9.0.0-beta.2.pre-237-g969db6452 (Git Commit Hash: 969db6452afbf0ffde65013520f01d8e4fac917d), with TiDB f40ad1d845.
The code of the cause is the same on master f3b7883955 (2026-10-28).

Lingua principale
C++
Stelle
1k
Fork
428
Merge medio
2g 7h
PR unite (30g)
100

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pingcap/tiflash

Tutte le issue di pingcap/tiflash

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.