JVM SIGSEGV in vframe::java_sender via allocation-profiler JVMTI GetStackTrace on a virtual thread (JDK 25, Alpine/musl) — regression in 1.62.0 (1.60.1 OK)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- java
- Domain
- observability-sre
Research direction
Start with the attached hs_err_pid1.log and trace the allocation-profiler path through Profiler::recordJVMTISample, ObjectSampler::recordAllocation, and JVMTI GetStackTrace on virtual threads. Compare behavior between tracer versions 1.60.1 and 1.62.0 on JDK 25 with Alpine/musl; done means allocation profiling no longer produces the reported SIGSEGV.
Written by the indexing model from the issue text.
Description
Tracer Version(s)
1.62.0 (crashes). Reverting to 1.60.1 resolves it — so this is a regression introduced between 1.60.1 and 1.62.0.
Java Version(s)
OpenJDK Runtime Environment Temurin-25.0.1+8 (build 25.0.1+8-LTS), linux-amd64
JVM Vendor
Eclipse Adoptium / Temurin
OS / Environment
Alpine Linux v3.22 (musl libc), QEMU 4 cores / 11G, container -Xmx6g, G1 GC. Spring Boot 4.0.6 app, virtual threads in use. Continuous profiler enabled (native ddprof, allocation profiling on).
Bug Report
After upgrading the agent to 1.62.0, the JVM hard-crashes (SIGSEGV, is_crash) intermittently — ~10 min after startup under normal traffic. The crash is in the Datadog allocation profiler: on a sampled allocation it calls JVMTI GetStackTrace, and walking a virtual thread's frames segfaults inside vframe::java_sender().
This appears to be a sibling of #9830 but on a different stackwalk path: #9830 was the async/ASGCT path (vframeStreamForte::forte_next), mitigated by -Ddd.profiling.ddprof.cstack=vm (default since 1.55.0). This one is the JVMTI GetStackTrace path used by the allocation sampler on virtual threads, which cstack=vm does not appear to cover (we are on 1.62.0, well past that default, and still crashing).
si_code: 128 (SI_KERNEL), si_addr: 0x0; the faulting thread is a virtual-thread carrier (ForkJoinPool-1-worker), _thread_in_vm. The ArrayList.grow / QueryExecutorImpl.processResults Java frames are just the innocent allocation being sampled (no large allocation — verified, the underlying collection is small).
Workaround: downgrade to 1.60.1 (clean). We expect DD_PROFILING_ALLOCATION_ENABLED=false (or DD_PROFILING_DDPROF_ENABLED=false, given musl) would also avoid it.
Crash stack (top frames from hs_err_pid1.log):
# SIGSEGV (0xb) at pc=..., pid=1, tid=197
# Java VM: OpenJDK 64-Bit Server VM Temurin-25.0.1+8 (25.0.1+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x1190128] vframe::java_sender() const+0x38
Current thread: JavaThread "ForkJoinPool-1-worker-18" daemon [_thread_in_vm, id=197]
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so] vframe::java_sender() const+0x38
V [libjvm.so] JvmtiEnvBase::get_stack_trace(javaVFrame*, int, int, jvmtiFrameInfo*, int*)
V [libjvm.so] GetStackTraceClosure::do_vthread(Handle)
V [libjvm.so] JvmtiHandshake::execute(...)
V [libjvm.so] JvmtiEnv::GetStackTrace(...)
V [libjvm.so] jvmti_GetStackTrace
C [libjavaProfiler-dd-*.so] Profiler::recordJVMTISample(...)
C [libjavaProfiler-dd-*.so] ObjectSampler::recordAllocation(...)
C [libjavaProfiler-dd-*.so] ObjectSampler::SampledObjectAlloc(...)
V [libjvm.so] JvmtiExport::post_sampled_object_alloc(JavaThread*, oopDesc*)
V [libjvm.so] MemAllocator::allocate() / InstanceKlass::allocate_objArray(...)
V [libjvm.so] OptoRuntime::new_array_C(...)
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.util.ArrayList.grow() java.base@25.0.1
J org.postgresql.core.v3.QueryExecutorImpl.processResults(...)
J jdk.internal.vm.Continuation.run() java.base@25.0.1
J java.lang.VirtualThread.runContinuation() java.base@25.0.1
J java.util.concurrent.ForkJoinPool.runWorker(...) java.base@25.0.1
j java.util.concurrent.ForkJoinWorkerThread.run() java.base@25.0.1
siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0
I can attach the full hs_err_pid1.log (and the .jfr) if useful.
- Dominant language
- Java
- Stars
- 737
- Forks
- 361
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 173
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from DataDog/dd-trace-java
-
type: feature request
Difficulty 1/5 1-3 hours Newbie friendliness 70/100
DataDog/dd-trace-java#10245 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 62/100
DataDog/dd-trace-java#12608 ·
-
type: bug report
Difficulty 4/5 3-5 days Newbie friendliness 35/100
DataDog/dd-trace-java#12597 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
DataDog/dd-trace-java#12540 · 3 comments · 1 assignee ·
-
Difficulty 3/5 1-2 days Newbie friendliness 25/100
DataDog/dd-trace-java#12480 ·
All issues in DataDog/dd-trace-java
Similar issues
-
awaiting triage bug Causes friction Hop Gui P1 P2 Transforms
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/flink-agents#1152 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
jenkinsci/blueocean-plugin#5417 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
objectionary/eo-graphs#75 ·