Iceberg: support add_files and replace_files (register existing data files into a table)

Open
  • Is something wrong?
#128,228 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, sql
Domain
databases

Research direction

The feature adds ALTER TABLE t EXECUTE add_files/replace_files for Iceberg tables. Start from the EXECUTE statement handling in ClickHouse's SQL parser and the Iceberg engine integration. Done means the commands register existing files with correct statistics and return metric rows.

Written by the indexing model from the issue text.

Description

comp-datalake

Describe the problem

There is no way to make an Iceberg table aware of data files that already exist in object storage. Every row has to be written through the sink of this engine, so a pipeline that produces Parquet files by another route (a clickhouse local run, file() output, another engine, a copy from a staging bucket) can not promote them into a table without re-reading and re-writing every byte.

The Iceberg answer to this is add_files / rewrite_files: register existing files in one snapshot, computing their statistics and partition values from the files themselves. Without it, every such pipeline needs an external Iceberg library just to publish its output.

Proposal

An ALTER TABLE ... EXECUTE entry point in the shape iceberg-java and pyiceberg use:

ALTER TABLE t EXECUTE add_files(paths [, check_duplicate_files = 1] [, allow_missing_columns = 0])
ALTER TABLE t EXECUTE replace_files(remove = [...] [, add = [...] ] [, relocate = 0] [, allow_missing_columns = 0])

Semantics:

  • paths is a path or an array of paths; a path ending in / means every *.parquet below it (hidden path components, starting with ., skipped). A relative path is relative to the table root. Each file is read once in this server to derive its record count, column sizes, value/null/NaN counts, bounds, and its partition tuple.
  • The partition tuple must resolve to a single partition of the current default spec. A file that spans partitions is not registered; the error names the file and the offending column.
  • add_files writes one snapshot with operation append. Duplicate detection (check_duplicate_files) compares against the files already live in the current snapshot.
  • replace_files removes the listed live files and adds the listed new ones in one snapshot with operation overwrite (or delete when there is no add). With relocate = 1 the i-th added file is a byte-identical copy of the i-th removed one (same size is verified): its manifest entry is copied with only the path changed, and the file is not read - which makes a storage-tier move a metadata operation instead of a data copy.
  • allow_missing_columns controls whether a file that lacks a column present in the table schema is registered (values then come from the column default) or refused.
  • On a v3 table every registered file is assigned a first_row_id, so _row_id works for rows that were never written by this engine.
  • The statement returns one row per metric (metric_name, metric_value) so the caller can see how many files and rows were registered.

Why EXECUTE rather than a new statement

The command operates on an external catalog object and reports metrics, which is what the EXECUTE surface is for. It also keeps the semantics of the operation separate from a plain INSERT, which has its own commit path.

Notes

  • Statistics have to be computed with the write.metadata.metrics.* of the table properties, or the manifest entry of a promoted file disagrees with a natively written one and pruning behaves differently for the two.
  • replace_files with relocate = 1 must verify the size, not just trust the caller: a manifest entry whose file_size_in_bytes is wrong reads the next file short.
  • The duplicate check is on the full path resolved against the table root, not the argument as typed, or the same file registers twice under two spellings.
  • Registering a file that is still being written by another process is not detectable here; that is the contract of the caller, but the error text for a truncated file should say so rather than reporting a parse failure.
Dominant language
C++
Stars
50.6k
Forks
9.2k
Avg merge
1d 2h
Merged PRs (30d)
524

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ClickHouse/ClickHouse

All issues in ClickHouse/ClickHouse

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.