delta-io/delta

[Kernel] Remove unecessary N.json loading for InCommitTimestamp value

Open

#4,914 建立於 2025年7月10日

在 GitHub 查看
 (3 留言) (0 反應) (0 負責人)Scala (8,807 star) (2,100 fork)batch import
good first issuekernel

描述

Today, Kernel will issue a cloud read request to read the Snapshot's latest N.json file, in order to read the in-commit-timestamp value from the CommitInfo action.

We should be able to remove such a cloud call (in 99.99% of cases). The basis for this is the following:

  • To get a Snapshot, we must resolve the latest Protocol and Metadata
  • We can get the latest P & M in one of two ways
  • First, by reading a .crc file. If so, the ICT value should be in there. (If not, we have to make the N.json cloud request later on)
  • Second, by reading the delta log .json files until we find the latest P & M. Even if the latest P & M are found in the latest N.json file, when ICT is enabled we require that the CommitInfo action is the first action written into such a file -- so we can guarantee we find it either way

貢獻者指南

技術棧
scala
領域
performance
議題類型
performance
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
half day
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
fresh
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
Familiarity with Delta Kernel snapshot resolutionUnderstanding of CommitInfo and N.jsonBasic Scala programming
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
70
研究方向
Investigate the snapshot resolution code in Delta Kernel to understand how the latest Protocol and Metadata are resolved. Look at the flow for reading .crc files and delta log .json files. Identify the specific call that loads N.json for InCommitTimestamp and modify it to first check the .crc file. Ensure changes maintain correctness for the 0.01% case where .crc does not have the ICT value.
[Kernel] Remove unecessary N.json loading for InCommitTimestamp value · delta-io/delta#4914 | Good First Issue