Iceberg `OPTIMIZE` sums all live files into `added-records`: `count()` inflated

Open
  • Is something wrong?
#128,265 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp
Domain
databases

Research direction

The summation bug is at src/Storages/ObjectStorage/DataLakes/Iceberg/Compaction.cpp:1117-1126, passed to MetadataGenerator.cpp:91. Read compactIcebergTable and setSnapshotTotals to see how snapshot_id_to_data_files accumulates. Change added_records to a delta of new files vs parent and widen to 64-bit. Done when the reproducer's trivial count() returns 2 after OPTIMIZE.

Written by the indexing model from the issue text.

Description

ai_p1 comp-datalake
Describe what's wrong
  • After OPTIMIZE on an Iceberg table, trivial count() returns more rows than a full scan.
  • Trigger: two consecutive appends plus a position delete, e.g. INSERT 2 rows, INSERT 1 row, DELETE 1 row, then OPTIMIZE.
  • Trivial count() should return 2 but returns 3, while the full scan correctly returns 2.
  • The wrong total-records is persisted in metadata, so system.tables.total_rows and every later trivial count stay wrong until the next rewrite.

Root cause: In Compaction.cpp:1117-1126 writeMetadataFiles sums new_records_count over every live file of the snapshot (snapshot_id_to_data_files) and passes it as added_records. setSnapshotTotals (MetadataGenerator.cpp:91) then adds it to the rewritten parent's total-records, so totals accumulate (1 + 2 = 3).

Analysis details (evidence, affected locations, impact)

Why we believe this is a bug: IcebergMetadata::optimize (IcebergMetadata.cpp:622) -> compactIcebergTable -> writeMetadataFiles (Compaction.cpp:1117-1126) -> MetadataGenerator::generateNextMetadata -> setSnapshotTotals (MetadataGenerator.cpp:91,100). The trivial count reads total-records back in IcebergMetadata::totalRows (IcebergMetadata.cpp:1751).

Affected locations:

Impact: Silent wrong count() on an Iceberg table after OPTIMIZE (allow_experimental_iceberg_compaction = 1, open-source builds). Any history with two consecutive append snapshots and at least one position delete hits it: with N appends the reported total is the sum of all prefix totals. added-records/total-records in the snapshot summaries are wrong for every reader of the metadata.

Does it reproduce on most recent release?

Confirmed on master (dcccc2b5920c); not checked on a release.

How to reproduce

Requires allow_insert_into_iceberg = 1, mutations_sync = 2 and allow_experimental_iceberg_compaction = 1; if the parent snapshot is delete-only the bug does not show.

# - no-fasttest: requires `IcebergLocal` (USE_AVRO build option)

# Trivial count() must match a full scan after OPTIMIZE of a table with two consecutive appends and a DELETE.

TABLE="t_default_${RANDOM}"
TABLE_PATH="/var/lib/clickhouse/user_files/${TABLE}/"

trap 'rm -rf "${TABLE_PATH}" 2>/dev/null' EXIT

clickhouse-client --query "CREATE TABLE ${TABLE} (x Int64) ENGINE = IcebergLocal('${TABLE_PATH}', 'Parquet')"
clickhouse-client --allow_insert_into_iceberg=1 --query "INSERT INTO ${TABLE} VALUES (1), (2)"
clickhouse-client --allow_insert_into_iceberg=1 --query "INSERT INTO ${TABLE} VALUES (3)"
clickhouse-client --allow_insert_into_iceberg=1 --mutations_sync=2 --query "ALTER TABLE ${TABLE} DELETE WHERE x = 1"
clickhouse-client --allow_experimental_iceberg_compaction=1 --query "OPTIMIZE TABLE ${TABLE}"

clickhouse-client --query "SELECT count() FROM ${TABLE} SETTINGS optimize_trivial_count_query = 1"
clickhouse-client --query "SELECT count() FROM ${TABLE} SETTINGS optimize_trivial_count_query = 0"

clickhouse-client --query "DROP TABLE ${TABLE} SYNC"

Verified: this block reproduces on current master (dcccc2b5920c) when pasted as is.

Expected behavior

Trivial count() after OPTIMIZE returns 2, matching the full scan.

Expected output of the reproducer above:

2
2
Error message and/or stacktrace

Trivial count() returns 3 because the snapshot's total-records counts the parent's rows twice.

Actual output of the reproducer above on master (dcccc2b5920c):

3
2
Suggested fix

Pass a real delta: the rows of the files the snapshot added (files not present in its parent), or total(snapshot) - total(rewritten parent). Make the accumulator 64-bit while there: Int32 total_records_count truncates once a snapshot holds more than 2^31 rows.

Additional context

Open risks:

  • Int32 total_records_count (Compaction.cpp:1117) also truncates the per-snapshot sum of UInt64 row counts past 2^31 rows, same lines.
  • Not introduced by #116941: the same summation and parent accumulation are in v26.3.9.8-lts and v26.8.9.10-lts. #116941 only lets OPTIMIZE through on more tables; its own new mixed-format test has the append, append, delete shape and checks rows with a full scan, so it does not see the wrong count.
  • A test merged into the suite needs the IS_CLOUD handling used by 04513/04815: on the Cloud build OPTIMIZE does not run this code and reports an exception instead.

Found during automated review of PR #116941; the bug predates that PR (it reproduces on the master build just before it merged), so the introducing change is not identified yet. Severity P1 · Finding h_pr116941_001

Dominant language
C++
Stars
50.6k
Forks
9.2k
Avg merge
1d 2h
Merged PRs (30d)
524

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ClickHouse/ClickHouse

All issues in ClickHouse/ClickHouse

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.