Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

`latest_sequence_number` is NULL for format v1 Iceberg snapshots instead of 0

Open Beginner friendly
  • Is something wrong?
#127,437 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp, sql
Domain
databases, testing

Research direction

Read src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp around set_latest_snapshot, then inspect src/Storages/System/StorageSystemIcebergMetadataLogEntries.cpp and the existing 05385_system_iceberg_metadata_log_entries.sh test. Reproduce with an Iceberg format v1 table and check how snapshots without a sequence-number are handled. Done when v1 snapshots report 0 and rows without a snapshot remain NULL, with the regression covered by the existing test.

Written by the indexing model from the issue text.

Description

ai_p2 comp-datalake
Describe what's wrong
  • system.iceberg_metadata_log_entries shows latest_sequence_number = NULL for every row with a latest_snapshot_id on a format v1 Iceberg table.
  • Trigger: create a table with iceberg_format_version = 1 and insert rows, since v1 snapshot JSON never carries a sequence-number key.
  • Spec and Spark's metadata_log_entries give 0 for v1, and ClickHouse's own system.iceberg_files.sequence_number also reports 0 for the same snapshots.
  • Joining latest_sequence_number to system.iceberg_files.sequence_number fails to match for any v1 table.

Root cause: In IcebergMetadata.cpp the set_latest_snapshot lambda only fills latest_sequence_number when the snapshot JSON has a sequence-number key, so for v1 snapshots the optional stays empty and StorageSystemIcebergMetadataLogEntries::fillData inserts NULL instead of defaulting to 0 as the spec requires.

Analysis details (evidence, affected locations, impact)

Why we believe this is a bug: StorageSystemIcebergMetadataLogEntries::fillData (StorageSystemIcebergMetadataLogEntries.cpp:82) calls IcebergMetadata::getMetadataLogEntries. Its set_latest_snapshot lambda (IcebergMetadata.cpp:1380) fills latest_sequence_number only when the snapshot JSON has sequence-number. v1 snapshots never carry that key, so the optional stays empty and fillData inserts NULL (StorageSystemIcebergMetadataLogEntries.cpp:102-105).

Affected locations:

Impact: Any format v1 table gets NULL where Spark gives 0: legacy Spark/Iceberg tables (v1 was the default before Iceberg 1.4) and tables created with iceberg_format_version = 1. A row that names a snapshot looks like it has no sequence number. Joining latest_sequence_number to system.iceberg_files.sequence_number finds no match for v1. Table data and reads are not affected.

Does it reproduce on most recent release?

Confirmed on master (4feb890db6ce); not checked on a release.

How to reproduce

Create an IcebergLocal table with iceberg_format_version = 1, insert two rows, then compare system.iceberg_metadata_log_entries against system.iceberg_files.

# A format v1 snapshot has sequence number 0: system.iceberg_metadata_log_entries must agree with system.iceberg_files.

CLICKHOUSE_CLIENT=$(echo clickhouse-client | sed 's/'"--send_logs_level=${CLICKHOUSE_CLIENT_SERVER_LOGS_LEVEL}"'/--send_logs_level=fatal/g')

ICEBERG_TABLE_PATH="/var/lib/clickhouse/user_files/lakehouses/default_t_v1"

rm -rf "${ICEBERG_TABLE_PATH}"

clickhouse-client --query "
    SET allow_experimental_insert_into_iceberg = 1;
    CREATE TABLE t_v1 (id Int32) ENGINE = IcebergLocal('${ICEBERG_TABLE_PATH}', 'Parquet') SETTINGS iceberg_format_version = 1;
    INSERT INTO t_v1 VALUES (1);
    INSERT INTO t_v1 VALUES (2);
"

clickhouse-client --query "
    SELECT latest_snapshot_id IS NULL, latest_schema_id, latest_sequence_number
    FROM system.iceberg_metadata_log_entries
    WHERE database = currentDatabase() AND table = 't_v1'
    ORDER BY timestamp;
"

clickhouse-client --query "
    SELECT count(), countIf(sequence_number = 0)
    FROM system.iceberg_files
    WHERE database = currentDatabase() AND table = 't_v1';
"

clickhouse-client --query "DROP TABLE t_v1;"

rm -rf "${ICEBERG_TABLE_PATH}"

Verified: this block reproduces on current master (4feb890db6ce) when pasted as is.

Expected behavior

For a format v1 snapshot, latest_sequence_number should read 0, matching the spec, Spark, and system.iceberg_files.

Expected output of the reproducer above:

0	0	0
0	0	0
2	2
Error message and/or stacktrace

For a format v1 snapshot with a latest_snapshot_id, latest_sequence_number reads NULL.

Actual output of the reproducer above on master (4feb890db6ce):

0	0	\N
0	0	\N
2	2
Suggested fix

Once a latest snapshot is found, default latest_sequence_number to 0 when the snapshot has no sequence-number. Rows written before any snapshot stay NULL. Add the v1 case to the existing 05385_system_iceberg_metadata_log_entries.sh rather than a new file.

Additional context

Open risks:

  • A table upgraded from v1 to v2 keeps its pre-upgrade snapshots without sequence-number. Rows whose latest snapshot predates the upgrade get the same NULL (not exercised).

Found during automated review of PR #125639; whether that PR introduced it could not be established, so nobody is tagged. Severity P2 · Finding h_pr125639_001

Dominant language
C++
Stars
50.5k
Forks
9.1k
Avg merge
20h 1m
Merged PRs (30d)
625

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ClickHouse/ClickHouse

All issues in ClickHouse/ClickHouse

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.