[Spark 4.2 UT][VL] Honor the per-read ignoreMissingFiles option in the native scan (SPARK-55857)

Aperta
#13,433 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, scala
Ambito
backend

Direzione di ricerca

Read GlutenConfig.scala, cpp/velox/utils/ConfigExtractor.cc, SubstraitToVeloxPlan.cc and BasicScanExecTransformer to trace how the session conf reaches Velox, then look at FileSplitReader.cpp in Velox for where the per-table override would go. Re-enable one excluded test in VeloxTestSettings.scala (e.g. GlutenParquetV1QuerySuite) and run it to reproduce the mismatch. Done means the per-read option takes precedence over the session conf in native scans, the error class matches Spark's, and the four excludes are removed — likely requiring upstream Velox changes, so it needs project familiarity.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Part of #13181 (sub-theme #13180). Found while re-enabling the Spark 4.2 UTs disabled in #13163.

Problem

The Velox native scan honors only the session conf spark.sql.files.ignoreMissingFiles. The per-read data source option ignoreMissingFiles (Spark >= 3.4) never reaches native code, so:

SQLConf read option Spark Gluten (native)
true (unset) skips missing file skips (OK)
false true skips missing file fails (Velox FILE_NOT_FOUND)
false (unset) FAILED_READ_FILE.FILE_NOT_EXIST fails with wrapped Velox error (different error class)
true false FAILED_READ_FILE.FILE_NOT_EXIST silently skips the missing file (wrong result)

Disabled tests (VeloxTestSettings.scala, spark42): SPARK-55857: Enabling/disabling ignoreMissingFiles in GlutenOrcV1QuerySuite, GlutenOrcV2QuerySuite, GlutenParquetV1QuerySuite, GlutenParquetV2QuerySuite.

Root cause
  • Gluten sends only the session conf: GlutenConfig.scala adds spark.sql.files.ignoreMissingFiles to the native session conf; cpp/velox/utils/ConfigExtractor.cc maps it to HiveConfig::kIgnoreMissingFilesSession.
  • Velox reads it only from session properties (velox/connectors/hive/FileSplitReader.cpp, fileConfig_->ignoreMissingFiles(connectorQueryCtx_->sessionProperties())), so it can't differ per scan.
  • Spark resolves it per scan via FileSourceOptions.ignoreMissingFiles (option, falling back to SQLConf) in FileScanRDD (V1) and FilePartitionReader (V2).
Proposed fix
  1. Velox: let a per-table parameter override the session value in FileSplitReader (e.g. tableHandle_->tableParameters()["ignore_missing_files"], else the session property).
  2. Gluten: pass new FileSourceOptions(options).ignoreMissingFiles from FileSourceScanExecTransformer / BatchScanExecTransformer and set the table parameter in SubstraitToVeloxPlan.cc (HiveTableHandle construction).
  3. Map the native missing-file error to Spark's FAILED_READ_FILE.FILE_NOT_EXIST (or add a Gluten test variant for that assertion only), then remove the 4 excludes.

Interim option (narrow, Gluten-only): fall back to vanilla scan in BasicScanExecTransformer validation when the read option differs from the session conf. This fixes the silent-skip correctness issue without a Velox change.

Lingua principale
Scala
Stelle
1.6k
Fork
685
Merge medio
4g 16h
PR unite (30g)
84

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/gluten

Tutte le issue di apache/gluten

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.