Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Deadlock: DynamicCodeGenerated allocates under _stubs_lock while the sampling signal handler spins in findRuntimeStub

Aberta
  • Algo errado?
#843 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
52/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
cpp
Domínio
performance

Direção de pesquisa

Start with ddprof-lib/src/main/cpp/hotspot/jitCodeCache.cpp, focusing on DynamicCodeGenerated and findRuntimeStub; the issue describes the lock/allocation cycle and two possible fix directions. Reproduce or inspect the relevant locking and allocation paths, then add or run tests covering signal-handler lookup while the stub lock is held. Done means the deadlock is avoided without unbounded spinning; the issue names no specific test file.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Summary

The native profiler (ddprof) deadlocks the JVM. A thread that holds the JitCodeCache stub lock calls malloc, while the profiler sampling signal interrupts another thread inside malloc, and that signal handler waits for the stub lock. All other sampled threads then spin in JitCodeCache::findRuntimeStub at 100% CPU, and the JVM never reaches a safepoint (jcmd cannot attach, the service stops answering).

We saw it twice in 4 hours on 2026-10-07, on 2 hosts, with the same cycle each time. We had seen the same symptom (N cores at 100%, JVM unresponsive, no log line) every few hours on this service before we captured stacks, on x86_64 and on arm64 (Graviton) hosts. Both captures below are from x86_64 hosts.

Versions

  • dd-java-agent 1.65.1, which loads ddprof 1.49.0 (libjavaProfiler)
  • Profiling on (-Ddd.profiling.enabled=true), default profiler settings (CPU and wall-clock samplers)
  • Amazon Corretto 25.0.0+36 (JDK 25), x86_64, G1 GC
  • Amazon Linux 2023, glibc malloc, kernel 6.1
  • The application calls a native library (libvips, through glib) from many threads, so threads are often inside malloc.

JitCodeCache::DynamicCodeGenerated and JitCodeCache::findRuntimeStub in ddprof-lib/src/main/cpp/hotspot/jitCodeCache.cpp are the same in v_1.49.0, v_1.51.0 and main (checked on 2026-10-07), so we think the newer versions are affected too.

The cycle

Thread Holds Waits for
JVM Service Thread, posting a deferred DynamicCodeGenerated event JitCodeCache::_stubs_lock (exclusive, taken in DynamicCodeGenerated) the glibc malloc lock (malloc from CodeCache::add)
A worker thread interrupted by the profiler signal inside malloc (_int_malloc / malloc_consolidate) the glibc malloc lock _stubs_lock: the signal handler calls findRuntimeStub → SpinLock::lockShared(), which spins with no bound
Every other sampled thread — _stubs_lock, spinning in lockShared()

DynamicCodeGenerated calls _runtime_stubs.add() (which allocates) while it holds _stubs_lock, and findRuntimeStub takes the same lock from the signal handler with the unbounded lockShared(). Possible fixes: do not allocate under _stubs_lock (prepare the entry first), or use the bounded tryLockShared(max_spins) in findRuntimeStub when it runs in a signal handler and give up the frame.

Stacks, capture 1 (6 threads at 100% CPU, 5 of them in findRuntimeStub)

Collected with gdb -p <pid> -batch -ex "thread apply all bt". Addresses and paths removed.

Thread "Service Thread":
#0  __lll_lock_wait_private ()
#1  malloc ()
#2  CodeCache::add(void const*, int, char const*, bool) ()
#3  JitCodeCache::DynamicCodeGenerated(_jvmtiEnv*, char const*, void const*, int) ()
#4  JvmtiExport::post_dynamic_code_generated_internal(char const*, void const*, void const*) ()
#5  JvmtiDeferredEvent::post() ()
#6  ServiceThread::service_thread_entry(JavaThread*, JavaThread*) ()
#7  JavaThread::thread_main_inner() [clone .part.0] ()
#8  Thread::call_run() ()
#9  thread_native_entry(Thread*) ()
#10 start_thread ()
#11 clone3 ()

Thread "<HTTP worker thread>":
#0  JitCodeCache::findRuntimeStub(void const*) ()
#1  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, void const*, unsigned long, unsigned long, int, bool*) ()
#2  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, int, bool*) ()
#3  HotspotSupport::walkJavaStack(StackWalkRequest&) ()
#4  Profiler::recordSample(void*, unsigned long long, int, int, unsigned long long, Event*, unsigned long long*) ()
#5  CTimer::signalHandler(int, siginfo_t*, void*) ()
#6  <signal handler called> ()
#7  malloc_consolidate ()
#8  _int_malloc ()
#9  calloc ()
#10 g_malloc0 ()
#11 g_type_create_instance ()
#12 g_object_new_internal ()
#13 g_object_new_with_properties ()
   ...

Thread "<HTTP worker thread>":
#0  JitCodeCache::findRuntimeStub(void const*) ()
#1  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, void const*, unsigned long, unsigned long, int, bool*) ()
#2  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, int, bool*) ()
#3  HotspotSupport::walkJavaStack(StackWalkRequest&) ()
#4  Profiler::recordSample(void*, unsigned long long, int, int, unsigned long long, Event*, unsigned long long*) ()
#5  CTimer::signalHandler(int, siginfo_t*, void*) ()
#6  <signal handler called> ()
#7  hwy::x86::Cpuid (abcd=..., count=..., level=...)
#8  hwy::x86::FlagsFromCPUID ()
   ...

Stacks, capture 2 (8 threads at 100% CPU, all 8 in findRuntimeStub)

Thread "Service Thread":
#0  __lll_lock_wait_private ()
#1  malloc ()
#2  CodeCache::add(void const*, int, char const*, bool) ()
#3  JitCodeCache::DynamicCodeGenerated(_jvmtiEnv*, char const*, void const*, int) ()
#4  JvmtiExport::post_dynamic_code_generated_internal(char const*, void const*, void const*) ()
#5  JvmtiDeferredEvent::post() ()
#6  ServiceThread::service_thread_entry(JavaThread*, JavaThread*) ()
#7  JavaThread::thread_main_inner() [clone .part.0] ()
#8  Thread::call_run() ()
#9  thread_native_entry(Thread*) ()
#10 start_thread ()
#11 clone3 ()

Thread "<HTTP worker thread>":
#0  JitCodeCache::findRuntimeStub(void const*) ()
#1  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, void const*, unsigned long, unsigned long, int, bool*) ()
#2  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, int, bool*) ()
#3  HotspotSupport::walkJavaStack(StackWalkRequest&) ()
#4  Profiler::recordSample(void*, unsigned long long, int, int, unsigned long long, Event*, unsigned long long*) ()
#5  CTimer::signalHandler(int, siginfo_t*, void*) ()
#6  <signal handler called> ()
#7  _int_malloc ()
#8  calloc ()
#9  g_malloc0 ()
#10 g_closure_new_simple ()
#11 g_cclosure_new ()
#12 g_signal_connect_data ()
#13 ??? ()
   ...

top -H on both hosts showed the worker threads at 99.9% CPU each, while the VM Thread was sleeping (state S, under 4% CPU). jcmd <pid> Thread.print failed with AttachNotSupportedException ... doesn't respond within 10500ms.

We worked around it with DD_PROFILING_DDPROF_ENABLED=false (JFR-based profiling). We can test a fix build if useful.

Linguagem predominante
C++
Estrelas
33
Forks
14
Merge médio
2d 12h
PRs com merge (30d)
41

Preparar o ambiente

  • Sem Dockerfile nem arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de DataDog/java-profiler

Todas as issues de DataDog/java-profiler

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.