[Spark 4.2 UT][VL] Honor the per-read ignoreMissingFiles option in the native scan (SPARK-55857)

Abierto
#13,433 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp, scala
Área
backend

Línea de trabajo

Read GlutenConfig.scala, cpp/velox/utils/ConfigExtractor.cc, SubstraitToVeloxPlan.cc and BasicScanExecTransformer to trace how the session conf reaches Velox, then look at FileSplitReader.cpp in Velox for where the per-table override would go. Re-enable one excluded test in VeloxTestSettings.scala (e.g. GlutenParquetV1QuerySuite) and run it to reproduce the mismatch. Done means the per-read option takes precedence over the session conf in native scans, the error class matches Spark's, and the four excludes are removed — likely requiring upstream Velox changes, so it needs project familiarity.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Part of #13181 (sub-theme #13180). Found while re-enabling the Spark 4.2 UTs disabled in #13163.

Problem

The Velox native scan honors only the session conf spark.sql.files.ignoreMissingFiles. The per-read data source option ignoreMissingFiles (Spark >= 3.4) never reaches native code, so:

SQLConf read option Spark Gluten (native)
true (unset) skips missing file skips (OK)
false true skips missing file fails (Velox FILE_NOT_FOUND)
false (unset) FAILED_READ_FILE.FILE_NOT_EXIST fails with wrapped Velox error (different error class)
true false FAILED_READ_FILE.FILE_NOT_EXIST silently skips the missing file (wrong result)

Disabled tests (VeloxTestSettings.scala, spark42): SPARK-55857: Enabling/disabling ignoreMissingFiles in GlutenOrcV1QuerySuite, GlutenOrcV2QuerySuite, GlutenParquetV1QuerySuite, GlutenParquetV2QuerySuite.

Root cause
  • Gluten sends only the session conf: GlutenConfig.scala adds spark.sql.files.ignoreMissingFiles to the native session conf; cpp/velox/utils/ConfigExtractor.cc maps it to HiveConfig::kIgnoreMissingFilesSession.
  • Velox reads it only from session properties (velox/connectors/hive/FileSplitReader.cpp, fileConfig_->ignoreMissingFiles(connectorQueryCtx_->sessionProperties())), so it can't differ per scan.
  • Spark resolves it per scan via FileSourceOptions.ignoreMissingFiles (option, falling back to SQLConf) in FileScanRDD (V1) and FilePartitionReader (V2).
Proposed fix
  1. Velox: let a per-table parameter override the session value in FileSplitReader (e.g. tableHandle_->tableParameters()["ignore_missing_files"], else the session property).
  2. Gluten: pass new FileSourceOptions(options).ignoreMissingFiles from FileSourceScanExecTransformer / BatchScanExecTransformer and set the table parameter in SubstraitToVeloxPlan.cc (HiveTableHandle construction).
  3. Map the native missing-file error to Spark's FAILED_READ_FILE.FILE_NOT_EXIST (or add a Gluten test variant for that assertion only), then remove the 4 excludes.

Interim option (narrow, Gluten-only): fall back to vanilla scan in BasicScanExecTransformer validation when the read option differs from the session conf. This fixes the silent-skip correctness issue without a Velox change.

Lenguaje dominante
Scala
Estrellas
1.6k
Forks
685
Merge medio
4 d 16 h
PR fusionados (30 d)
84

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/gluten

Todos los issues de apache/gluten

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.