[VL] Segmentation fault when dumping task for micro benchmark

Open
#13,225 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
15/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
cpp, java, scala

Research direction

Start from the native frame AlignedBuffer::freeToPool() in libvelox.so, reached through ColumnarBatchOutIterator.nativeClose, VeloxIteratorApi and IteratorsV1.PayloadCloser, which is where the close path runs during task teardown. The issue has no reproduction query or Gluten version, so first reproduce a scan + filter + project task and confirm the SIGSEGV. Done means the crash no longer occurs under that query and the root cause in the buffer lifetime is identified.

Written by the indexing model from the issue text.

Description

bug triage
Backend

VL (Velox)

Bug description

Encountered SIGSEGV when dumping a simple scan + filter + project task

Current thread (0x00007f341f012000):  JavaThread "Executor task launch worker for task 22.0 in stage 36.0 (TID 1097)" daemon [_thread_in_native, id=2183, stack(0x00007f33cb4fe000,0x00007f33cb5fe000)]

Stack: [0x00007f33cb4fe000,0x00007f33cb5fe000],  sp=0x00007f33cb5f9a20,  free space=1006k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libvelox.so+0x2393cef]  facebook::velox::AlignedBuffer::freeToPool()+0x1f

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 13266  org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeClose(J)V (0 bytes) @ 0x00007f349de82060 [0x00007f349de82020+0x0000000000000040]
J 19654 c1 org.apache.gluten.backendsapi.velox.VeloxIteratorApi$$Lambda$2259+0x00007f3440c2f198.apply$mcV$sp()V (16 bytes) @ 0x00007f3495f9ea14 [0x00007f3495f9e820+0x00000000000001f4]
J 20304 c2 org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext()Z (14 bytes) @ 0x00007f349e313f50 [0x00007f349e313980+0x00000000000005d0]
J 17252 c2 org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext()Z (23 bytes) @ 0x00007f349e166a0c [0x00007f349e1669c0+0x000000000000004c]
J 19477 c1 org.apache.gluten.execution.VeloxColumnarToRowExec$$anon$1.hasNext()Z (10 bytes) @ 0x00007f3497818324 [0x00007f34978182c0+0x0000000000000064]
J 16654 c2 scala.collection.Iterator$$anon$10.hasNext()Z (10 bytes) @ 0x00007f349e0bcde8 [0x00007f349e0bcda0+0x0000000000000048]
J 18928 c1 scala.collection.TraversableOnce$FlattenOps$$anon$2.hasNext()Z (52 bytes) @ 0x00007f34976e18d4 [0x00007f34976e17a0+0x0000000000000134]
J 20303 c2 org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes) @ 0x00007f349e312e08 [0x00007f349e312d40+0x00000000000000c8]
J 21527 c1 org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(ZILscala/collection/Iterator;)Lscala/collection/Iterator; (311 bytes) @ 0x00007f34978eb2ec [0x00007f34978eae00+0x00000000000004ec]
J 21535 c1 org.apache.spark.sql.execution.SparkPlan$$Lambda$3587+0x00007f34410cd9f8.apply(Ljava/lang/Object;)Ljava/lang/Object; (16 bytes) @ 0x00007f34978f4074 [0x00007f34978f3fc0+0x00000000000000b4]
J 19406 c1 org.apache.spark.rdd.RDD$$Lambda$1060+0x00007f34406b5f38.apply(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (17 bytes) @ 0x00007f34977f6204 [0x00007f34977f60c0+0x0000000000000144]
J 19492 c1 org.apache.spark.rdd.MapPartitionsRDD.compute(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator; (36 bytes) @ 0x00007f349781cc94 [0x00007f349781bb40+0x0000000000001154]
J 18631 c1 org.apache.spark.rdd.RDD.iterator(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator; (44 bytes) @ 0x00007f349762647c [0x00007f34976248a0+0x0000000000001bdc]
J 19269 c1 org.apache.spark.scheduler.ResultTask.runTask(Lorg/apache/spark/TaskContext;)Ljava/lang/Object; (210 bytes) @ 0x00007f34977977ac [0x00007f3497796ec0+0x00000000000008ec]
J 21624 c1 org.apache.spark.TaskContext.runTaskWithListeners(Lorg/apache/spark/scheduler/Task;)Ljava/lang/Object; (74 bytes) @ 0x00007f349792f294 [0x00007f349792f220+0x0000000000000074]
J 20347 c1 org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;ILscala/collection/immutable/Map;Lscala/Option;)Ljava/lang/Object; (356 bytes) @ 0x00007f3496878d04 [0x00007f3496875f60+0x0000000000002da4]
J 20411 c1 org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Lorg/apache/spark/executor/Executor$TaskRunner;Lscala/runtime/BooleanRef;)Ljava/lang/Object; (54 bytes) @ 0x00007f3496f328ac [0x00007f3496f32660+0x000000000000024c]
J 20410 c1 org.apache.spark.executor.Executor$TaskRunner$$Lambda$892+0x00007f344059b260.apply()Ljava/lang/Object; (12 bytes) @ 0x00007f34971e1344 [0x00007f34971e12c0+0x0000000000000084]
J 19717 c1 org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object; (211 bytes) @ 0x00007f3496376f8c [0x00007f3496376f20+0x000000000000006c]
J 20328 c1 org.apache.spark.executor.Executor$TaskRunner.run()V (2830 bytes) @ 0x00007f3497870f7c [0x00007f349786c2e0+0x0000000000004c9c]
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 [email protected]
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 [email protected]
j  java.lang.Thread.run()V+11 [email protected]
v  ~StubRoutines::call_stub
Gluten version

No response

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

Dominant language
Scala
Stars
1.6k
Forks
685
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.