Support Iceberg `SparkCopyOnWriteScan` in `IcebergReflection.ICEBERG_SCAN_CLASSES`
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 86/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- scala
- Domain
- backend, data-engineering
Research direction
Open IcebergReflection.scala and trace how CometScanRule uses ICEBERG_SCAN_CLASSES to identify Iceberg scans. Add recognition for org.apache.iceberg.spark.source.SparkCopyOnWriteScan, then verify that Copy-On-Write MERGE, UPDATE, and DELETE scans are recognized for native Comet conversion.
Written by the indexing model from the issue text.
Description
What is the problem the feature request solves?
When running Copy-On-Write (CoW) table operations such as MERGE INTO, UPDATE, or DELETE against Iceberg tables in Spark Iceberg's Spark planning extension creates a
SparkCopyOnWriteScan (org.apache.iceberg.spark.source.SparkCopyOnWriteScan) for scanning the target table data files.
Found the gap while testing native iceberg writes #4487
Currently, IcebergReflection.ICEBERG_SCAN_CLASSES only includes:
• org.apache.iceberg.spark.source.SparkBatchQueryScan
• org.apache.iceberg.spark.source.SparkStagedScan
Because SparkCopyOnWriteScan is missing from ICEBERG_SCAN_CLASSES, IcebergReflection.isIcebergScanClass(...) returns false when CometScanRule evaluates a physical plan
containing a SparkCopyOnWriteScan. As a result, Comet fails to recognize SparkCopyOnWriteScan as a supported Iceberg scan and falls back to JVM Spark execution for the scan
phase of Copy-On-Write queries.
Describe the potential solution
Add SparkCopyOnWriteScan ("org.apache.iceberg.spark.source.SparkCopyOnWriteScan") to IcebergReflection.ClassNames and include it in IcebergReflection.ICEBERG_SCAN_CLASSES.
// In IcebergReflection.scala
object ClassNames {
...
val SPARK_COPY_ON_WRITE_SCAN = "org.apache.iceberg.spark.source.SparkCopyOnWriteScan"
}
val ICEBERG_SCAN_CLASSES: Set[String] =
Set(
ClassNames.SPARK_BATCH_QUERY_SCAN,
ClassNames.SPARK_STAGED_SCAN,
ClassNames.SPARK_COPY_ON_WRITE_SCAN)
This enables CometScanRule to recognize SparkCopyOnWriteScan instances and convert them to native Comet scans during Iceberg Copy-On-Write MERGE, UPDATE, and DELETE
operations.
Additional context
SparkCopyOnWriteScan extends Iceberg's standard Spark scan primitives (SparkBatchQueryScan / SparkScan) and behaves identically with respect to file tasks, table schemas, and
filter pushdowns during data file reads.
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 377
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 244
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion-comet
-
area:ci enhancement requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion-comet#6078 ·
-
area:ci bug priority:low
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion-comet#6060 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
apache/datafusion-comet#6028 ·
-
enhancement
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
apache/datafusion-comet#5861 ·
-
requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/datafusion-comet#5661 ·
All issues in apache/datafusion-comet
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
ergoplatform/ergodocs#614 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching length Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
chipsalliance/rocket-chip#3831 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100