Cached UC managed Delta table reads remain stale after RTAS or overwrite on Spark 4.0.1
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start by reading Spark 4.0.1's DataSourceV2Strategy and the cached-plan invalidation path, then compare the qualified-name invalidation change in PR #52599. Reproduce the RTAS and saveAsTable overwrite cases with the supplied Spark, Delta, and UC versions, checking whether cache invalidation matches the cached V1 LogicalRelation. Done means fresh reads return replacement rows and the dependent cached view is refreshed, with the append control still fresh.
由索引模型根据 Issue 内容生成。
描述
We observed stale cached query results after successful replacement of a UC managed Delta table on Spark 4.0.1 with Delta 4.2.0 and OSS Unity Catalog 0.6.0. Both SQL REPLACE TABLE AS SELECT and DataFrameWriter.saveAsTable(mode="overwrite") are affected. The writes and reads occur in the same Spark session; this is not an external writer requiring explicit refresh.
Reproduction workflow
Precondition: a configured UC catalog supporting managed Delta tables, Java 17, and the matching Spark-line Delta/UC artifacts. Use independent tables for the two variants.
- Create a UC managed Delta table containing
(1, 'original'), withdelta.feature.catalogManaged=supported, deletion vectors disabled and CDF disabled. - Cache the qualified table name and materialize its rows. For the RTAS variant, also cache and materialize a dependent temporary view.
- Replace the table with
(2, 'replacement'), (3, 'replacement')through RTAS, or overwrite the independently seeded table with(3, 'overwritten')through saveAsTable. - Read the qualified table name through a newly created query, without REFRESH TABLE, CLEAR CACHE or manual uncache.
Expected: the new query returns the replacement rows; the dependent cached view also reflects the replacement.
Actual on Spark 4.0.1: Delta history advances to version 1, but the fresh query still returns (1, 'original'). The dependent view in the RTAS variant is stale too. No persisted-data loss has been observed.
Verification
| Spark | Delta | Execution | RTAS | saveAsTable overwrite | Append control |
|---|---|---|---|---|---|
| 4.0.1 | 4.2.0 | local[4] | Stale | Stale | Fresh |
| 4.0.1 | 4.2.0 | Standalone, separate driver/executor JVMs | Stale | Stale | Fresh |
| 4.1.1 | 4.2.0 | local[4] | Fresh | Fresh | Fresh |
The Spark-line comparison uses the corresponding Delta and UC artifacts, not byte-identical dependencies.
Diagnosis and question
The cached analyzed plan contains a V1 LogicalRelation. Spark 4.0.1's DataSourceV2Strategy constructs a synthetic DataSourceV2Relation for invalidation; it does not match the cached plan (sameResult == false, cache lookup empty). Both writes execute AtomicReplaceTableAsSelectExec. Repeating relation-based uncache after recording the stale read leaves the cache intact; qualified-name uncache clears it and the next read returns the committed replacement rows.
PR #52599, whose stated purpose is TimeTravelSpec support, also changes this strategy to qualified-name invalidation. We reference that code change as a possible explanation for the passing Spark 4.1.1 control, not as confirmation that this specific defect was reported or fixed there. We have not tested an isolated backport. Please confirm ownership of the V1/V2 invalidation mismatch and whether a focused correction for Spark 4.0.x is appropriate.
- 主要语言
- Scala
- 星标
- 44.2k
- 派生
- 29.5k
- 平均合并
- 8 小时 43 分钟
- 30 天内合并 PR
- 1
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/spark 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 67/100
维护者通常 2 天内回复
-
[SQL][CONNECT] SparkConnectClient may drop metadata (auth headers) on retried/reattached RPCs if metadata is a non-list iterable可能已有人在做 @zainmirzaaa 于 62 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 2 天内回复
-
[CORE] bug report: collision of resource profile id可能已有人在做 @ybapat 于 87 天前认领。 未关闭
难度 1/5 1 小时以内 新手友好度 82/100
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 2 天内回复
-
Merge spark PR script hard blocks on JIRA可能已有人在做 @holdenk 于 1 天前认领。 未关闭
apache/spark#59693 · 已指派 1 人 ·
维护者通常 2 天内回复
相似的 Issue
-
enhancement good first issue help wanted
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 72/100
scylladb/scylla-migrator#417 ·
-
[Docs] Document `spark.databricks.delta.optimize.maxThreads` in Compaction (bin-packing) section未关闭delta-docs delta-table-maintenance documentation
难度 1/5 1-3 小时 新手友好度 85/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 66/100
software-mansion/starknet-jvm#641 ·