[VL] Native Iceberg scan silently drops rows at random; a plain Parquet read of the same files is exact

Aperta
#13,570 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
aws, java, scala, spark

Direzione di ricerca

Start by tracing IcebergScanTransformer and the Iceberg split information passed to Velox’s Iceberg split reader; compare that path with the working plain Parquet scan. Reproduce the intermittent row loss with the supplied query and Iceberg table setup, then inspect split planning and scan execution for where rows are omitted. Done means native Iceberg scans match vanilla Spark across repeated runs without silent row loss.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Backend

VL (Velox)

Bug description

With the native Iceberg scan (IcebergScanTransformer), a simple count over one partition of an Iceberg table returns fewer rows than vanilla Spark in most runs. The number of missing rows changes from run to run (0 to 543 here). No task fails, no task is retried, and the logs have no error or warning.

In the same engine, with the same query:

  • spark.gluten.sql.columnar.batchscan=false (vanilla Iceberg reader, rest of the plan in Gluten) is exact every time.
  • The same 15 data files, read as plain Parquet with the native FileSourceScanExecTransformer (parquet.`<dir>/{file1,...,file15}` ), are exact every time. The result is identical to vanilla Spark on the same path, 5 runs out of 5.

So the rows seem to be lost in the Iceberg split path (Gluten's Iceberg split info or Velox's Iceberg split reader). Parquet decoding, S3 reads, filters and aggregation return correct results on the plain Parquet path.

We found it in production-like tests: an Iceberg overwrite of about 8.2 billion rows committed 49 to 398 fewer rows than vanilla Spark in 4 runs out of 5, with all tasks successful.

Query (table and value names changed):

SELECT count(*) AS n, count(DISTINCT id) AS ids, bit_xor(xxhash64(id)) AS id_hash
FROM db.events
WHERE name IN (<27 event names>)
  AND server_created_at >= CAST('2026-11-04' AS DATE) AND server_created_at < CAST('2026-11-05' AS DATE)
  AND app_id = '<one app>' AND platform = 'android'

Expected (vanilla Spark, and Gluten with batchscan=false): n = 39,125,071.

Table

  • Iceberg 1.10.0, Glue catalog, S3FileIO, data on S3. Partition spec: day(server_created_at), bucket(app_id).
  • No delete files in the partition (position or equality).
  • The Velox Iceberg doc marks partitioned tables as "Fallback mostly". Here the scan of a table partitioned by day and bucket is offloaded (IcebergScanTransformer), and it returns wrong results without a fallback or an error.
  • The matching rows are in 15 data files of one partition (the partition has 83 data files, 292 M rows, 19.4 GB).
  • In our first analysis, all lost rows came from one file: 3,668,268 rows, 520 MB, 5 row groups, split_offsets = [4, 132840796, 259353067, 386848326, 513486252]. With the default 128 MB split size, Iceberg plans this file as several file scan tasks. But a split size of 1 GB, so that no file is cut into several tasks, does not stop the loss (see below).

Results (same query, executors with 3 cores, 5g heap, 10g off-heap)

Setup Runs Missing rows per run
1.8.0-SNAPSHOT nightly (2026-10-27), native Iceberg scan 11 −76, −211, −18, −433, −32, −38, −1, 0, −80, −64, −38
same, prefetchRowGroups=0 5 −66, −64, 0, −543, −85
same, maxCoalescedDistance=0B 5 0, −122, −9, −57, −320
same, loadQuantum=8MB 5 −62, −270, −203, −112, 0
same, fileHandleCacheEnabled=false 5 −61, −152, −402, −9, −245
same, IOThreads=0 and SplitPreloadPerDriver=0 (the 8.2 B row job) 3 −398, −146, −49
same, read option split-size = 1 GB (no file is cut: 4 tasks instead of 39) 5 −420, −416, −63, −423, −142 (control in the same application, default split size: −104, 0, −324, −70, −195)
1.7.0-SNAPSHOT nightly (2026-06-18), Graviton3 nodes only 5 0, −212, −323, −38, −18
1.8.0-SNAPSHOT, batchscan=false 3 0, 0, 0
1.8.0-SNAPSHOT, the same 15 files as plain Parquet, native scan 5 0, 0, 0, 0, 0 (identical to vanilla Spark)

(The Velox settings are static configs, so each of them ran in its own Spark application.)

Other observations

  • The lost rows come in fixed groups: two runs on two different applications lost the same 9 rows (same count, same id_hash). A 1.7.0-SNAPSHOT run and a 1.8.0-SNAPSHOT run lost the same 38 rows. Only which groups go missing changes between runs.
  • The lost rows are close together: most of them are in a narrow range of user_id values.
  • The issue is in 1.7.0-SNAPSHOT (June 2026) too, so it is not a recent regression. We did not find a related fix on main up to the 2026-11-06 nightly.
  • On the plain Parquet path, the count differs from the Iceberg count by 7,083 rows also with Gluten off. We did not investigate this difference: the comparison above is always Gluten against vanilla Spark on the same path.
  • Possibly related: #12545 (silent data loss when preload threads fail). But IOThreads=0 does not fix our case.
Gluten version

main branch

(Nightly jars from nightlies.apache.org/gluten/nightly-release-jdk17: gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.8.0-SNAPSHOT.jar built 2026-10-27T02:24:53Z, sha256 f7b1b71a849a56662284e5f5670baaa75c0a1c4a64dbac20415e6e6908b1e166, and ...-1.7.0-SNAPSHOT.jar built 2026-06-18T01:03:52Z, sha256 05795d60495b901b3888561240440d528ffecee8dc57af4439fac8cd5b22008a.)

Spark version

Spark-3.5.x

Spark configurations
spark.plugins=org.apache.gluten.GlutenPlugin
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=10g
spark.executor.memory=5g
spark.executor.memoryOverhead=6g
spark.executor.cores=3
spark.driver.defaultJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
spark.executor.defaultJavaOptions=-XX:+UseTransparentHugePages -Dio.netty.tryReflectionSetAccessible=true -XX:TrimNativeHeapInterval=5000
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.<cat>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<cat>.type=glue
spark.sql.catalog.<cat>.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.shuffle.partitions=1000
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.minExecutors=80
spark.dynamicAllocation.maxExecutors=200
spark.decommission.enabled=true
spark.storage.decommission.enabled=true
System information
  • Apache Spark 3.5.6 on Kubernetes (EKS), Iceberg 1.10.0 (iceberg-spark-runtime-3.5_2.12), Java 17 (Temurin 17.0.15), Ubuntu 22.04 base image.
  • linux/aarch64: AWS Graviton executors (r6gd, r7gd, r8gd). The 1.7.0-SNAPSHOT runs used r7gd only, because that arm64 jar needs SVE.
  • All runs are on linux/aarch64, with the official nightly jars (the releases have no aarch64 jar). We could not test x86. The plain Parquet read on the same aarch64 jar is exact.
Relevant logs

No error or warning. All tasks succeed.

Lingua principale
Scala
Stelle
1.6k
Fork
685
Merge medio
4g 16h
PR unite (30g)
84

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/gluten

Tutte le issue di apache/gluten

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.