[VL] Segmentation fault when dumping task for micro benchmark
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 15/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Active
- Domain
- data-engineering
Research direction
Start from the native frame AlignedBuffer::freeToPool() in libvelox.so, reached through ColumnarBatchOutIterator.nativeClose, VeloxIteratorApi and IteratorsV1.PayloadCloser, which is where the close path runs during task teardown. The issue has no reproduction query or Gluten version, so first reproduce a scan + filter + project task and confirm the SIGSEGV. Done means the crash no longer occurs under that query and the root cause in the buffer lifetime is identified.
Written by the indexing model from the issue text.
Description
Backend
VL (Velox)
Bug description
Encountered SIGSEGV when dumping a simple scan + filter + project task
Current thread (0x00007f341f012000): JavaThread "Executor task launch worker for task 22.0 in stage 36.0 (TID 1097)" daemon [_thread_in_native, id=2183, stack(0x00007f33cb4fe000,0x00007f33cb5fe000)]
Stack: [0x00007f33cb4fe000,0x00007f33cb5fe000], sp=0x00007f33cb5f9a20, free space=1006k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libvelox.so+0x2393cef] facebook::velox::AlignedBuffer::freeToPool()+0x1f
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 13266 org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeClose(J)V (0 bytes) @ 0x00007f349de82060 [0x00007f349de82020+0x0000000000000040]
J 19654 c1 org.apache.gluten.backendsapi.velox.VeloxIteratorApi$$Lambda$2259+0x00007f3440c2f198.apply$mcV$sp()V (16 bytes) @ 0x00007f3495f9ea14 [0x00007f3495f9e820+0x00000000000001f4]
J 20304 c2 org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext()Z (14 bytes) @ 0x00007f349e313f50 [0x00007f349e313980+0x00000000000005d0]
J 17252 c2 org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext()Z (23 bytes) @ 0x00007f349e166a0c [0x00007f349e1669c0+0x000000000000004c]
J 19477 c1 org.apache.gluten.execution.VeloxColumnarToRowExec$$anon$1.hasNext()Z (10 bytes) @ 0x00007f3497818324 [0x00007f34978182c0+0x0000000000000064]
J 16654 c2 scala.collection.Iterator$$anon$10.hasNext()Z (10 bytes) @ 0x00007f349e0bcde8 [0x00007f349e0bcda0+0x0000000000000048]
J 18928 c1 scala.collection.TraversableOnce$FlattenOps$$anon$2.hasNext()Z (52 bytes) @ 0x00007f34976e18d4 [0x00007f34976e17a0+0x0000000000000134]
J 20303 c2 org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes) @ 0x00007f349e312e08 [0x00007f349e312d40+0x00000000000000c8]
J 21527 c1 org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(ZILscala/collection/Iterator;)Lscala/collection/Iterator; (311 bytes) @ 0x00007f34978eb2ec [0x00007f34978eae00+0x00000000000004ec]
J 21535 c1 org.apache.spark.sql.execution.SparkPlan$$Lambda$3587+0x00007f34410cd9f8.apply(Ljava/lang/Object;)Ljava/lang/Object; (16 bytes) @ 0x00007f34978f4074 [0x00007f34978f3fc0+0x00000000000000b4]
J 19406 c1 org.apache.spark.rdd.RDD$$Lambda$1060+0x00007f34406b5f38.apply(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (17 bytes) @ 0x00007f34977f6204 [0x00007f34977f60c0+0x0000000000000144]
J 19492 c1 org.apache.spark.rdd.MapPartitionsRDD.compute(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator; (36 bytes) @ 0x00007f349781cc94 [0x00007f349781bb40+0x0000000000001154]
J 18631 c1 org.apache.spark.rdd.RDD.iterator(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator; (44 bytes) @ 0x00007f349762647c [0x00007f34976248a0+0x0000000000001bdc]
J 19269 c1 org.apache.spark.scheduler.ResultTask.runTask(Lorg/apache/spark/TaskContext;)Ljava/lang/Object; (210 bytes) @ 0x00007f34977977ac [0x00007f3497796ec0+0x00000000000008ec]
J 21624 c1 org.apache.spark.TaskContext.runTaskWithListeners(Lorg/apache/spark/scheduler/Task;)Ljava/lang/Object; (74 bytes) @ 0x00007f349792f294 [0x00007f349792f220+0x0000000000000074]
J 20347 c1 org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;ILscala/collection/immutable/Map;Lscala/Option;)Ljava/lang/Object; (356 bytes) @ 0x00007f3496878d04 [0x00007f3496875f60+0x0000000000002da4]
J 20411 c1 org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Lorg/apache/spark/executor/Executor$TaskRunner;Lscala/runtime/BooleanRef;)Ljava/lang/Object; (54 bytes) @ 0x00007f3496f328ac [0x00007f3496f32660+0x000000000000024c]
J 20410 c1 org.apache.spark.executor.Executor$TaskRunner$$Lambda$892+0x00007f344059b260.apply()Ljava/lang/Object; (12 bytes) @ 0x00007f34971e1344 [0x00007f34971e12c0+0x0000000000000084]
J 19717 c1 org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object; (211 bytes) @ 0x00007f3496376f8c [0x00007f3496376f20+0x000000000000006c]
J 20328 c1 org.apache.spark.executor.Executor$TaskRunner.run()V (2830 bytes) @ 0x00007f3497870f7c [0x00007f349786c2e0+0x0000000000004c9c]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 [email protected]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j java.lang.Thread.run()V+11 [email protected]
v ~StubRoutines::call_stub
Gluten version
No response
Spark version
None
Spark configurations
No response
System information
No response
Relevant logs
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 685
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day