[Spark 4.2 UT][VL] Honor the per-read ignoreMissingFiles option in the native scan (SPARK-55857)
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
Read GlutenConfig.scala, cpp/velox/utils/ConfigExtractor.cc, SubstraitToVeloxPlan.cc and BasicScanExecTransformer to trace how the session conf reaches Velox, then look at FileSplitReader.cpp in Velox for where the per-table override would go. Re-enable one excluded test in VeloxTestSettings.scala (e.g. GlutenParquetV1QuerySuite) and run it to reproduce the mismatch. Done means the per-read option takes precedence over the session conf in native scans, the error class matches Spark's, and the four excludes are removed — likely requiring upstream Velox changes, so it needs project familiarity.
Written by the indexing model from the issue text.
Description
Part of #13181 (sub-theme #13180). Found while re-enabling the Spark 4.2 UTs disabled in #13163.
Problem
The Velox native scan honors only the session conf spark.sql.files.ignoreMissingFiles. The per-read data source option ignoreMissingFiles (Spark >= 3.4) never reaches native code, so:
| SQLConf | read option | Spark | Gluten (native) |
|---|---|---|---|
| true | (unset) | skips missing file | skips (OK) |
| false | true |
skips missing file | fails (Velox FILE_NOT_FOUND) |
| false | (unset) | FAILED_READ_FILE.FILE_NOT_EXIST |
fails with wrapped Velox error (different error class) |
| true | false |
FAILED_READ_FILE.FILE_NOT_EXIST |
silently skips the missing file (wrong result) |
Disabled tests (VeloxTestSettings.scala, spark42): SPARK-55857: Enabling/disabling ignoreMissingFiles in GlutenOrcV1QuerySuite, GlutenOrcV2QuerySuite, GlutenParquetV1QuerySuite, GlutenParquetV2QuerySuite.
Root cause
- Gluten sends only the session conf:
GlutenConfig.scalaaddsspark.sql.files.ignoreMissingFilesto the native session conf;cpp/velox/utils/ConfigExtractor.ccmaps it toHiveConfig::kIgnoreMissingFilesSession. - Velox reads it only from session properties (
velox/connectors/hive/FileSplitReader.cpp,fileConfig_->ignoreMissingFiles(connectorQueryCtx_->sessionProperties())), so it can't differ per scan. - Spark resolves it per scan via
FileSourceOptions.ignoreMissingFiles(option, falling back to SQLConf) inFileScanRDD(V1) andFilePartitionReader(V2).
Proposed fix
- Velox: let a per-table parameter override the session value in
FileSplitReader(e.g.tableHandle_->tableParameters()["ignore_missing_files"], else the session property). - Gluten: pass
new FileSourceOptions(options).ignoreMissingFilesfromFileSourceScanExecTransformer/BatchScanExecTransformerand set the table parameter inSubstraitToVeloxPlan.cc(HiveTableHandleconstruction). - Map the native missing-file error to Spark's
FAILED_READ_FILE.FILE_NOT_EXIST(or add a Gluten test variant for that assertion only), then remove the 4 excludes.
Interim option (narrow, Gluten-only): fall back to vanilla scan in BasicScanExecTransformer validation when the read option differs from the session conf. This fixes the silent-skip correctness issue without a Velox change.
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day