[Spark 4.2 UT][VL] Honor the per-read ignoreMissingFiles option in the native scan (SPARK-55857)

Open
#13,433 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, scala
Domain
backend

Research direction

Read GlutenConfig.scala, cpp/velox/utils/ConfigExtractor.cc, SubstraitToVeloxPlan.cc and BasicScanExecTransformer to trace how the session conf reaches Velox, then look at FileSplitReader.cpp in Velox for where the per-table override would go. Re-enable one excluded test in VeloxTestSettings.scala (e.g. GlutenParquetV1QuerySuite) and run it to reproduce the mismatch. Done means the per-read option takes precedence over the session conf in native scans, the error class matches Spark's, and the four excludes are removed — likely requiring upstream Velox changes, so it needs project familiarity.

Written by the indexing model from the issue text.

Description

Part of #13181 (sub-theme #13180). Found while re-enabling the Spark 4.2 UTs disabled in #13163.

Problem

The Velox native scan honors only the session conf spark.sql.files.ignoreMissingFiles. The per-read data source option ignoreMissingFiles (Spark >= 3.4) never reaches native code, so:

SQLConf read option Spark Gluten (native)
true (unset) skips missing file skips (OK)
false true skips missing file fails (Velox FILE_NOT_FOUND)
false (unset) FAILED_READ_FILE.FILE_NOT_EXIST fails with wrapped Velox error (different error class)
true false FAILED_READ_FILE.FILE_NOT_EXIST silently skips the missing file (wrong result)

Disabled tests (VeloxTestSettings.scala, spark42): SPARK-55857: Enabling/disabling ignoreMissingFiles in GlutenOrcV1QuerySuite, GlutenOrcV2QuerySuite, GlutenParquetV1QuerySuite, GlutenParquetV2QuerySuite.

Root cause
  • Gluten sends only the session conf: GlutenConfig.scala adds spark.sql.files.ignoreMissingFiles to the native session conf; cpp/velox/utils/ConfigExtractor.cc maps it to HiveConfig::kIgnoreMissingFilesSession.
  • Velox reads it only from session properties (velox/connectors/hive/FileSplitReader.cpp, fileConfig_->ignoreMissingFiles(connectorQueryCtx_->sessionProperties())), so it can't differ per scan.
  • Spark resolves it per scan via FileSourceOptions.ignoreMissingFiles (option, falling back to SQLConf) in FileScanRDD (V1) and FilePartitionReader (V2).
Proposed fix
  1. Velox: let a per-table parameter override the session value in FileSplitReader (e.g. tableHandle_->tableParameters()["ignore_missing_files"], else the session property).
  2. Gluten: pass new FileSourceOptions(options).ignoreMissingFiles from FileSourceScanExecTransformer / BatchScanExecTransformer and set the table parameter in SubstraitToVeloxPlan.cc (HiveTableHandle construction).
  3. Map the native missing-file error to Spark's FAILED_READ_FILE.FILE_NOT_EXIST (or add a Gluten test variant for that assertion only), then remove the 4 excludes.

Interim option (narrow, Gluten-only): fall back to vanilla scan in BasicScanExecTransformer validation when the read option differs from the session conf. This fixes the silent-skip correctness issue without a Velox change.

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.