delta-io/delta

[Kernel] Remove unecessary N.json loading for InCommitTimestamp value

Open

#4,914 opened on Jul 10, 2025

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Scala (8,807 stars) (2,100 forks)batch import
good first issuekernel

Description

Today, Kernel will issue a cloud read request to read the Snapshot's latest N.json file, in order to read the in-commit-timestamp value from the CommitInfo action.

We should be able to remove such a cloud call (in 99.99% of cases). The basis for this is the following:

  • To get a Snapshot, we must resolve the latest Protocol and Metadata
  • We can get the latest P & M in one of two ways
  • First, by reading a .crc file. If so, the ICT value should be in there. (If not, we have to make the N.json cloud request later on)
  • Second, by reading the delta log .json files until we find the latest P & M. Even if the latest P & M are found in the latest N.json file, when ICT is enabled we require that the CommitInfo action is the first action written into such a file -- so we can guarantee we find it either way

Contributor guide

Tech stack
scala
Domain
performance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Familiarity with Delta Kernel snapshot resolutionUnderstanding of CommitInfo and N.jsonBasic Scala programming
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Investigate the snapshot resolution code in Delta Kernel to understand how the latest Protocol and Metadata are resolved. Look at the flow for reading .crc files and delta log .json files. Identify the specific call that loads N.json for InCommitTimestamp and modify it to first check the .crc file. Ensure changes maintain correctness for the 0.01% case where .crc does not have the ICT value.