Row lineage

Open
  • Is something wrong?
#4,080 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data

Research direction

Start with the scan projection and REST scan-planning code, then inspect the manifest and data-file metadata handling for first_row_id and data sequence numbers. Check the existing tests around scans, deletes, and inspect.files() before adding coverage. Done means scans expose both lineage columns with the specified null handling and position semantics, and the requested integration comparisons pass.

Written by the indexing model from the issue text.

Description

V3
Feature Request / Improvement

We want to be able to read the row-lineage values on a table. This is a blocker for writing v3 tables.

table.scan(selected_fields=("id", "_row_id", "_last_updated_sequence_number"))
  • Data files with a null first_row_id inherit it from the manifest's first_row_id plus a running record count: #3972
  • Define the reserved columns _row_id (2147483540) and _last_updated_sequence_number (2147483539)
  • Let scans project them
  • Use the values stored in the file if there are any. Fill nulls with first_row_id + pos and the file's data sequence number.
  • Compute pos before applying deletes
  • Show first_row_id in inspect.files()
  • REST scan planning drops first_row_id
  • Integration test: each file's first_row_id matches Spark's MIN(_row_id): #3972
  • Integration test: Spark MERGE / UPDATE / rewrite_data_files, then compare with Spark's SELECT _row_id, _last_updated_sequence_number
Dominant language
Python
Stars
1.2k
Forks
664
Avg merge
2d 10h
Merged PRs (30d)
131

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-python

All issues in apache/iceberg-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.