[DISCUSSION] How should we test datafusion-spark against Spark?

Offen
  • Stimmt etwas nicht?
#27,103 15 Kommentare 3 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
25/100
Issue-Typ
Feature
Klarheit
Muss geklärt werden
Aktivitätsstatus
Aktiv
Tech-Stack
jupyter, rust, sql
Bereich
databases, testing

Rechercherichtung

Read the Spark compatibility tests in test_files/spark/*.slt and the linked discussions about live Spark audits, Spark version support, and physical-path coverage. Compare the three options in the issue, including their CI and downstream release effects. This is a discussion with no selected direction; progress means the community agrees on an approach or identifies another option.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

spark

At the DataFusion community call on 2026-11-04 there was renewed interest in datafusion-spark. This issue lists the challenges and some options for making it reliably Spark-compatible.

Challenges

  1. Development has stalled. In my opinion, most DataFusion reviewers aren't interested in Spark functionality, and the people who are, in Comet and Sail, rarely review or contribute because of the challenges below. Since November 2025, 79% of spark PRs came from outside the Comet and Sail regulars. Comet regulars reviewed 89 of the 347 PRs, 56 of them by one reviewer, and Sail contributors reviewed 5. Comet runs about 34 Spark expressions on this crate and about 96 on its own code, including at least 21 functions this crate also implements. It would like to upstream more, such as CAST (#11201), but that isn't practical without real tests that ensure Spark compatibility.
  2. We don't test against Spark. Nothing checks the expected results in test_files/spark/*.slt against Spark, and reviewers can't verify them without running Spark. Audits against a live Spark (#23893) keep finding divergences, including tests that assert the wrong value (#26333, #26338). I raised this risk when the crate moved here (#5600).
  3. Spark versions differ. Behavior changes between Spark releases (ANSI mode became the default in 4.0, for example), but the crate has no notion of a Spark version (#21698) and an .slt query can only have one expected result (#23887). Comet supports five Spark versions (3.4 to 4.2), and even Comet runs only one of them before merge, the rest nightly or on demand.
  4. Two kinds of users, one test path. Comet builds physical plans directly, so it needs invoke_with_args(). Projects that use DataFusion's planner and optimizer can use simplify() rewrites instead. SLT always goes through the planner and optimizer, so it tests the rewrite rather than the physical path, and functions with no physical implementation pass every test (#21959).

Options

  1. Run Spark in this repo's CI. #21508 was an earlier attempt. This catches divergences before merge, but a DataFusion PR could then fail a Spark test its author can't debug, and it adds a JVM and a Spark version matrix to CI whose usage we are already trying to reduce (#25148).
  2. Generate expected results with Spark and check them in per Spark version. A script runs the .slt queries on each supported Spark version and records the results, and CI runs only DataFusion against them. Nobody needs Spark to debug a failure, since it is an ordinary SLT diff against a value Spark produced. Whoever adds a test has to run the script (or ask a bot to), the queries must be valid Spark SQL (DataFusion now has a Spark dialect, #22529), and passing on more than one Spark version needs #21698.
  3. Move the crate to a new apache/datafusion-spark repository under the DataFusion PMC, like Comet, Ballista and datafusion-iceberg. Its CI can run real Spark across versions, and its contributors and reviewers would be people who can debug Spark. The pieces of core that only this crate uses would move with it: enable_ansi_mode, datafusion.spark.*, datafusion-cli --spark and the Spark handling in the sqllogictest runner.

These can be combined, and option 2 works in either repository. Each option also needs a way to test the physical path, for example by running the Spark .slt files a second time with simplify() disabled.

Impact on downstream projects

Comet and Sail depend on datafusion-spark (55.2.0), which today ships with each DataFusion release. Options 1 and 2 keep that. Option 3 would mean:

  • A separate release, with its own vote, after each DataFusion release before Comet and Sail can upgrade. In #5600 Sail said waiting on out-of-tree crates slowed its upgrades, but that this goes away if Comet and Sail maintainers help keep the crate in sync.
  • DataFusion API changes absorbed when the new repo upgrades, as Comet does today, rather than in the same PR.
  • Public APIs for code shared with datafusion-functions (#17964).

Which options, or combination, should we pursue? Are there others?

cc @alamb @comphead @mbutrovich @shehabgamin @linhr @Jefffrey @coderfender @xanderbailey

Vorherrschende Sprache
Rust
Sterne
9.6k
Forks
2.6k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
345

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/datafusion

Alle Issues in apache/datafusion

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.