[VL] Flaky Spark 4.2 UT: GlutenTransformWithStateUnsafeRowSuite "test that introducing TTL after restart fails query"
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start in gluten-ut/spark42/.../VeloxTestSettings.scala and inspect the GlutenTransformWithStateUnsafeRowSuite settings. Exclude tests with the prefix test that introducing TTL after restart fails query to stop the flaky cases from blocking CI. Run the Spark 4.2 test job and confirm both variants are excluded; the Spark failure investigation and re-enabling the tests are follow-up work.
Written by the indexing model from the issue text.
Description
Backend
VL (Velox)
Bug description
GlutenTransformWithStateUnsafeRowSuite (Spark 4.2) is flaky in the spark-test-spark42 (1) CI job. The failing test is:
test that introducing TTL after restart fails query (encoding = UnsafeRow) (with changelog checkpointing)test that introducing TTL after restart fails query (encoding = UnsafeRow) (without changelog checkpointing)
The two variants fail intermittently; which one fails changes from run to run. The test reports a failure with the exception it expects:
org.apache.spark.sql.execution.streaming.state.StateStoreValueSchemaNotCompatible: [STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE] Provided value schema does not match existing state value schema.
Please check number and type of fields.
Existing value_schema=StructType(StructField(value,LongType,false)) and new value_schema=StructType(StructField(value,StructType(StructField(value,LongType,false)),true),StructField(ttlExpirationMs,LongType,true)).
at org.apache.spark.sql.execution.streaming.state.StateStoreErrors$.stateStoreValueSchemaNotCompatible(StateStoreErrors.scala:170)
at org.apache.spark.sql.execution.streaming.state.StateSchemaCompatibilityChecker.check(StateSchemaCompatibilityChecker.scala:247)
...
Gluten doesn't execute any part of this query: every operator (MicroBatchScan, TransformWithState, Exchange, Sort, WriteToDataSourceV2, ...) falls back to vanilla Spark. The test expects this exception inside ExpectFailure[StateStoreValueSchemaNotCompatible], but sometimes the error appears to escape before ExpectFailure can catch it. This looks like a timing race in the Spark test itself, not a Gluten correctness problem.
Recent failures in unrelated PRs and in the scheduled main run:
- https://github.com/apache/gluten/actions/runs/40300608111 (PR #13453)
- https://github.com/apache/gluten/actions/runs/40339712752
- https://github.com/apache/gluten/actions/runs/40328474600
- https://github.com/apache/gluten/actions/runs/40286339268
- https://github.com/apache/gluten/actions/runs/40153084868 (scheduled
main)
Proposed action
Short term: exclude the test in gluten-ut/spark42/.../VeloxTestSettings.scala so it stops blocking CI:
enableSuite[GlutenTransformWithStateUnsafeRowSuite]
.excludeByPrefix("test that introducing TTL after restart fails query")
Follow-up: find out why the error escapes ExpectFailure (possibly made more likely by running suites in parallel JVMs), then either fix it or report it upstream to Spark, and re-enable the test.
Spark version
Spark 4.2.0
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 665
- Avg merge
- 4d 45m
- Merged PRs (30d)
- 111
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 2 days
-
[VL] ANSI workflow checks out main instead of the PRPossibly taken @rdtr claimed this 8 days ago. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 2 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthPossibly taken @jiangjiangtian claimed this 37 days ago. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 2 days
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/gluten#12711 · 2 comments · 1 reaction ·
Maintainers usually reply within 2 days
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 73/100
lichess-org/lila#22298 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Python: legacy module proxies emit DeprecationWarning on introspection, so inspect.stack() warns after `import glide`Possibly taken A pull request linked to this issue is open or already merged. Openbug 🐞 Untriaged user issue
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
valkey-io/valkey-glide#7443 ·
Maintainers usually reply within 1 day
-
OpenAPI: PowSolutions requires pk, w and d, but a V2 solution needs only nPossibly taken A pull request linked to this issue is open or already merged. OpenA-api C-bug S-ready-for-human
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
ergoplatform/ergo#2851 ·
Maintainers usually reply within 2 days
-
? - Needs Triage audit_4.2.0 feature request test
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
NVIDIA/cudf-spark#16614 ·
Maintainers usually reply within 1 day