Spark DataSource backed by a DataFusion TableProvider over ADBC
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 25/100
- Issue 类型
- 功能
- 描述清晰度
- 描述清楚
- 活跃度
- 停滞
调研方向
先从 #111 中引用的实现开始，然后将其与此 issue 声明的范围进行比较：adbc-datafusion DataSourceV2、pushdowns、分区读取、executor connection pooling 和 PySpark 覆盖。运行 issue 中提到的端到端覆盖，并验证列出的每项能力都已得到体现且正常工作。
由索引模型根据 Issue 内容生成。
描述
Is your feature request related to a problem or challenge?
Spark users want to read data from a DataFusion TableProvider as a native Spark DataSourceV2. Today there is no first-class path; options are either a bespoke per-operation JNI surface (more native surface to maintain) or copying data out of process.
Describe the solution you'd like
A Spark DataSourceV2 connector that places the native boundary at a standard ADBC driver. Spark talks to the upstream arrow-adbc Java driver manager (adbc-core + adbc-driver-jni), which loads a native DataFusion ADBC cdylib and returns arrow-java ArrowReaders consumed zero-copy as ArrowColumnVectors on the cluster-provided Arrow. This reuses the upstream ADBC bindings rather than reproducing them.
Scope:
adbc-datafusionformat registered as aDataSourceV2; schema probed on the driver.- Projection / filter / limit pushdown via Substrait, with a SQL fallback.
- Multi-partition reads (
executePartitioned/readPartition) and atarget_partitionsoption. - Per-executor connection pool to amortize driver/database setup across task slots.
- An example DataFusion ADBC driver cdylib plus end-to-end (PySpark) coverage.
Describe alternatives you've considered
A plain-C scan ABI + hand-written JNI shim (discussed on #103 / #104). The ADBC approach reuses standard, separately-reviewed bindings and a stable driver contract instead.
Additional context
Implemented in #111.
- 主要语言
- Java
- 星标
- 32
- 派生
- 12
- PR 合并指标
- 30 天内没有已合并 PR
贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/datafusion-java 的其他 Issue
-
难度 5/5 一周以上 新手友好度 35/100
apache/datafusion-java#116 ·
-
enhancement
难度 5/5 一周以上 新手友好度 42/100
apache/datafusion-java#96 ·
-
enhancement
难度 5/5 一周以上 新手友好度 38/100
apache/datafusion-java#95 ·
-
enhancement
难度 4/5 3-5 天 新手友好度 35/100
apache/datafusion-java#86 · 3 条评论 ·
-
enhancement
难度 5/5 一周以上 新手友好度 45/100
apache/datafusion-java#68 ·
查看 apache/datafusion-java 的全部 Issue
相似的 Issue
-
bug untriaged
难度 2/5 1-3 小时 新手友好度 84/100
opensearch-project/ml-commons#5094 ·
-
bug
难度 2/5 1-3 小时 新手友好度 85/100
-
emitter:client:csharp feature
难度 2/5 1-3 小时 新手友好度 72/100
-
affects/8.10 affects/8.9 component/clients kind/bug likelihood/mid severity/mid
难度 2/5 1-3 小时 新手友好度 78/100
-
bug frontend maui-pilot
难度 2/5 1-3 小时 新手友好度 72/100