[VL] Native Iceberg scan silently drops rows at random; a plain Parquet read of the same files is exact

Open
#13,570 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
aws, java, scala, spark

Research direction

Start by tracing IcebergScanTransformer and the Iceberg split information passed to Velox’s Iceberg split reader; compare that path with the working plain Parquet scan. Reproduce the intermittent row loss with the supplied query and Iceberg table setup, then inspect split planning and scan execution for where rows are omitted. Done means native Iceberg scans match vanilla Spark across repeated runs without silent row loss.

Written by the indexing model from the issue text.

Description

Backend

VL (Velox)

Bug description

With the native Iceberg scan (IcebergScanTransformer), a simple count over one partition of an Iceberg table returns fewer rows than vanilla Spark in most runs. The number of missing rows changes from run to run (0 to 543 here). No task fails, no task is retried, and the logs have no error or warning.

In the same engine, with the same query:

  • spark.gluten.sql.columnar.batchscan=false (vanilla Iceberg reader, rest of the plan in Gluten) is exact every time.
  • The same 15 data files, read as plain Parquet with the native FileSourceScanExecTransformer (parquet.`<dir>/{file1,...,file15}` ), are exact every time. The result is identical to vanilla Spark on the same path, 5 runs out of 5.

So the rows seem to be lost in the Iceberg split path (Gluten's Iceberg split info or Velox's Iceberg split reader). Parquet decoding, S3 reads, filters and aggregation return correct results on the plain Parquet path.

We found it in production-like tests: an Iceberg overwrite of about 8.2 billion rows committed 49 to 398 fewer rows than vanilla Spark in 4 runs out of 5, with all tasks successful.

Query (table and value names changed):

SELECT count(*) AS n, count(DISTINCT id) AS ids, bit_xor(xxhash64(id)) AS id_hash
FROM db.events
WHERE name IN (<27 event names>)
  AND server_created_at >= CAST('2026-11-04' AS DATE) AND server_created_at < CAST('2026-11-05' AS DATE)
  AND app_id = '<one app>' AND platform = 'android'

Expected (vanilla Spark, and Gluten with batchscan=false): n = 39,125,071.

Table

  • Iceberg 1.10.0, Glue catalog, S3FileIO, data on S3. Partition spec: day(server_created_at), bucket(app_id).
  • No delete files in the partition (position or equality).
  • The Velox Iceberg doc marks partitioned tables as "Fallback mostly". Here the scan of a table partitioned by day and bucket is offloaded (IcebergScanTransformer), and it returns wrong results without a fallback or an error.
  • The matching rows are in 15 data files of one partition (the partition has 83 data files, 292 M rows, 19.4 GB).
  • In our first analysis, all lost rows came from one file: 3,668,268 rows, 520 MB, 5 row groups, split_offsets = [4, 132840796, 259353067, 386848326, 513486252]. With the default 128 MB split size, Iceberg plans this file as several file scan tasks. But a split size of 1 GB, so that no file is cut into several tasks, does not stop the loss (see below).

Results (same query, executors with 3 cores, 5g heap, 10g off-heap)

Setup Runs Missing rows per run
1.8.0-SNAPSHOT nightly (2026-10-27), native Iceberg scan 11 −76, −211, −18, −433, −32, −38, −1, 0, −80, −64, −38
same, prefetchRowGroups=0 5 −66, −64, 0, −543, −85
same, maxCoalescedDistance=0B 5 0, −122, −9, −57, −320
same, loadQuantum=8MB 5 −62, −270, −203, −112, 0
same, fileHandleCacheEnabled=false 5 −61, −152, −402, −9, −245
same, IOThreads=0 and SplitPreloadPerDriver=0 (the 8.2 B row job) 3 −398, −146, −49
same, read option split-size = 1 GB (no file is cut: 4 tasks instead of 39) 5 −420, −416, −63, −423, −142 (control in the same application, default split size: −104, 0, −324, −70, −195)
1.7.0-SNAPSHOT nightly (2026-06-18), Graviton3 nodes only 5 0, −212, −323, −38, −18
1.8.0-SNAPSHOT, batchscan=false 3 0, 0, 0
1.8.0-SNAPSHOT, the same 15 files as plain Parquet, native scan 5 0, 0, 0, 0, 0 (identical to vanilla Spark)

(The Velox settings are static configs, so each of them ran in its own Spark application.)

Other observations

  • The lost rows come in fixed groups: two runs on two different applications lost the same 9 rows (same count, same id_hash). A 1.7.0-SNAPSHOT run and a 1.8.0-SNAPSHOT run lost the same 38 rows. Only which groups go missing changes between runs.
  • The lost rows are close together: most of them are in a narrow range of user_id values.
  • The issue is in 1.7.0-SNAPSHOT (June 2026) too, so it is not a recent regression. We did not find a related fix on main up to the 2026-11-06 nightly.
  • On the plain Parquet path, the count differs from the Iceberg count by 7,083 rows also with Gluten off. We did not investigate this difference: the comparison above is always Gluten against vanilla Spark on the same path.
  • Possibly related: #12545 (silent data loss when preload threads fail). But IOThreads=0 does not fix our case.
Gluten version

main branch

(Nightly jars from nightlies.apache.org/gluten/nightly-release-jdk17: gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.8.0-SNAPSHOT.jar built 2026-10-27T02:24:53Z, sha256 f7b1b71a849a56662284e5f5670baaa75c0a1c4a64dbac20415e6e6908b1e166, and ...-1.7.0-SNAPSHOT.jar built 2026-06-18T01:03:52Z, sha256 05795d60495b901b3888561240440d528ffecee8dc57af4439fac8cd5b22008a.)

Spark version

Spark-3.5.x

Spark configurations
spark.plugins=org.apache.gluten.GlutenPlugin
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=10g
spark.executor.memory=5g
spark.executor.memoryOverhead=6g
spark.executor.cores=3
spark.driver.defaultJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
spark.executor.defaultJavaOptions=-XX:+UseTransparentHugePages -Dio.netty.tryReflectionSetAccessible=true -XX:TrimNativeHeapInterval=5000
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.<cat>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<cat>.type=glue
spark.sql.catalog.<cat>.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.shuffle.partitions=1000
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.minExecutors=80
spark.dynamicAllocation.maxExecutors=200
spark.decommission.enabled=true
spark.storage.decommission.enabled=true
System information
  • Apache Spark 3.5.6 on Kubernetes (EKS), Iceberg 1.10.0 (iceberg-spark-runtime-3.5_2.12), Java 17 (Temurin 17.0.15), Ubuntu 22.04 base image.
  • linux/aarch64: AWS Graviton executors (r6gd, r7gd, r8gd). The 1.7.0-SNAPSHOT runs used r7gd only, because that arm64 jar needs SVE.
  • All runs are on linux/aarch64, with the official nightly jars (the releases have no aarch64 jar). We could not test x86. The plain Parquet read on the same aarch64 jar is exact.
Relevant logs

No error or warning. All tasks succeed.

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.