[DISCUSSION] How should we test datafusion-spark against Spark?

Read the Spark compatibility tests in test_files/spark/*.slt and the linked discussions about live Spark audits, Spark version support, and physical-path coverage. Compare the three options in the issue, including their CI and downstream release effects. This is a discussion with no selected direction; progress means the community agrees on an approach or identifies another option.

At the DataFusion community call on 2026-11-04 there was renewed interest in datafusion-spark. This issue lists the challenges and some options for making it reliably Spark-compatible.

Challenges

1. Development has stalled. In my opinion, most DataFusion reviewers aren't interested in Spark functionality, and the people who are, in Comet and Sail, rarely review or contribute because of the challenges below. Since November 2025, 79% of spark PRs came from outside the Comet and Sail regulars. Comet regulars reviewed 89 of the 347 PRs, 56 of them by one reviewer, and Sail contributors reviewed 5. Comet runs about 34 Spark expressions on this crate and about 96 on its own code, including at least 21 functions this crate also implements. It would like to upstream more, such as CAST (#11201), but that isn't practical without real tests that ensure Spark compatibility.
2. We don't test against Spark. Nothing checks the expected results in test_files/spark/*.slt against Spark, and reviewers can't verify them without running Spark. Audits against a live Spark (#23893) keep finding divergences, including tests that assert the wrong value (#26333, #26338). I raised this risk when the crate moved here (#5600).
3. Spark versions differ. Behavior changes between Spark releases (ANSI mode became the default in 4.0, for example), but the crate has no notion of a Spark version (#21698) and an .slt query can only have one expected result (#23887). Comet supports five Spark versions (3.4 to 4.2), and even Comet runs only one of them before merge, the rest nightly or on demand.
4. Two kinds of users, one test path. Comet builds physical plans directly, so it needs invoke_with_args(). Projects that use DataFusion's planner and optimizer can use simplify() rewrites instead. SLT always goes through the planner and optimizer, so it tests the rewrite rather than the physical path, and functions with no physical implementation pass every test (#21959).

Options

1. Run Spark in this repo's CI. #21508 was an earlier attempt. This catches divergences before merge, but a DataFusion PR could then fail a Spark test its author can't debug, and it adds a JVM and a Spark version matrix to CI whose usage we are already trying to reduce (#25148).
2. Generate expected results with Spark and check them in per Spark version. A script runs the .slt queries on each supported Spark version and records the results, and CI runs only DataFusion against them. Nobody needs Spark to debug a failure, since it is an ordinary SLT diff against a value Spark produced. Whoever adds a test has to run the script (or ask a bot to), the queries must be valid Spark SQL (DataFusion now has a Spark dialect, #22529), and passing on more than one Spark version needs #21698.
3. Move the crate to a new apache/datafusion-spark repository under the DataFusion PMC, like Comet, Ballista and datafusion-iceberg. Its CI can run real Spark across versions, and its contributors and reviewers would be people who can debug Spark. The pieces of core that only this crate uses would move with it: enable_ansi_mode, datafusion.spark.*, datafusion-cli --spark and the Spark handling in the sqllogictest runner.

These can be combined, and option 2 works in either repository. Each option also needs a way to test the physical path, for example by running the Spark .slt files a second time with simplify() disabled.

Impact on downstream projects

Comet and Sail depend on datafusion-spark (55.2.0), which today ships with each DataFusion release. Options 1 and 2 keep that. Option 3 would mean:

• A separate release, with its own vote, after each DataFusion release before Comet and Sail can upgrade. In #5600 Sail said waiting on out-of-tree crates slowed its upgrades, but that this goes away if Comet and Sail maintainers help keep the crate in sync.
• DataFusion API changes absorbed when the new repo upgrades, as Comet does today, rather than in the same PR.
• Public APIs for code shared with datafusion-functions (#17964).

Which options, or combination, should we pursue? Are there others?

cc @alamb @comphead @mbutrovich @shehabgamin @linhr @Jefffrey @coderfender @xanderbailey