Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

feature: record field lineage when a cast mints a new field ID

Open
  • Is something wrong?
#10,039 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
15/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
data

Research direction

This is design work, not a first-timer task. Start by reading #8658 for how non-reusable field IDs work, then the alter_columns cast path and the Transaction V2 DropField + AddField handling named in the issue. Settle the open questions first: a reserved schema metadata key versus a pb::Field proto field, and whether non-cast replacements also record derived_from. Done means a maintainer-agreed design, then an implementation with tests that the pointer is persisted and ignored for reads of old-ID data.

Written by the indexing model from the issue text.

Description

feature

Since #8658, a cast assigns the column a new field ID once non-reusable field IDs are active. That keeps stale data keyed by the old ID (index segments, MemWAL rows, overlays, unrewritten files) from being read as the new type, but it loses the link between the old and new field. Tools that track a column across versions (CDC consumers, catalogs, schema history) cannot tell that field 5 is field 0 after a cast.

Proposal: when a field replaces another, record the replaced ID on the new field, e.g. derived_from: 0. Casts form a chain that can be followed back through older versions.

  • Informational only. Readers must not use it to treat data, index segments, or WAL rows written for the old ID as valid for the new one.
  • Persist it in the schema, not only in the transaction, since transaction files are cleaned up. A reserved field metadata key survives legacy writers that rewrite the schema; an unknown pb::Field proto field would be dropped.
  • Record it only on datasets with non-reusable field IDs active. On legacy datasets the old ID may be reused, so the pointer would be ambiguous.
  • Set it from both alter_columns casts and Transaction V2 casts (DropField + AddField + data). V2 conflict detection can then use it as the explicit signal that an AddField replaces a dropped field.

Open questions:

  • Metadata key vs. pb::Field field.
  • Whether other replacements besides casts (e.g. a rewrite of a column under a new definition) should record it.
Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.