Spark DataSource backed by a DataFusion TableProvider over ADBC
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Stale
- Domain
- backend, data-engineering, distributed-systems
Research direction
Start with the implementation referenced in #111, then compare it with this issue's stated scope: the adbc-datafusion DataSourceV2, pushdowns, partitioned reads, executor connection pooling, and PySpark coverage. Run the end-to-end coverage mentioned in the issue and verify that each listed capability is represented and working.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem or challenge?
Spark users want to read data from a DataFusion TableProvider as a native Spark DataSourceV2. Today there is no first-class path; options are either a bespoke per-operation JNI surface (more native surface to maintain) or copying data out of process.
Describe the solution you'd like
A Spark DataSourceV2 connector that places the native boundary at a standard ADBC driver. Spark talks to the upstream arrow-adbc Java driver manager (adbc-core + adbc-driver-jni), which loads a native DataFusion ADBC cdylib and returns arrow-java ArrowReaders consumed zero-copy as ArrowColumnVectors on the cluster-provided Arrow. This reuses the upstream ADBC bindings rather than reproducing them.
Scope:
adbc-datafusionformat registered as aDataSourceV2; schema probed on the driver.- Projection / filter / limit pushdown via Substrait, with a SQL fallback.
- Multi-partition reads (
executePartitioned/readPartition) and atarget_partitionsoption. - Per-executor connection pool to amortize driver/database setup across task slots.
- An example DataFusion ADBC driver cdylib plus end-to-end (PySpark) coverage.
Describe alternatives you've considered
A plain-C scan ABI + hand-written JNI shim (discussed on #103 / #104). The ADBC approach reuses standard, separately-reviewed bindings and a stable driver contract instead.
Additional context
Implemented in #111.
- Dominant language
- Java
- Stars
- 32
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion-java
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/datafusion-java#116 ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 42/100
apache/datafusion-java#96 ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 38/100
apache/datafusion-java#95 ·
-
Create first release Openenhancement
Difficulty 4/5 3-5 days Newbie friendliness 35/100
apache/datafusion-java#86 · 3 comments ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 45/100
apache/datafusion-java#68 ·
All issues in apache/datafusion-java
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Two open-case totals on one screen: the Programs tile says 15,858 and the nav badge says 15,868 Openbug frontend maui-pilot
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
objectionary/eo-graphs#74 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100