Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Deadlock: DynamicCodeGenerated allocates under _stubs_lock while the sampling signal handler spins in findRuntimeStub

Ouverte
  • Quelque chose ne va pas ?
#843 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
52/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
cpp
Domaine
performance

Piste de recherche

Start with ddprof-lib/src/main/cpp/hotspot/jitCodeCache.cpp, focusing on DynamicCodeGenerated and findRuntimeStub; the issue describes the lock/allocation cycle and two possible fix directions. Reproduce or inspect the relevant locking and allocation paths, then add or run tests covering signal-handler lookup while the stub lock is held. Done means the deadlock is avoided without unbounded spinning; the issue names no specific test file.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Summary

The native profiler (ddprof) deadlocks the JVM. A thread that holds the JitCodeCache stub lock calls malloc, while the profiler sampling signal interrupts another thread inside malloc, and that signal handler waits for the stub lock. All other sampled threads then spin in JitCodeCache::findRuntimeStub at 100% CPU, and the JVM never reaches a safepoint (jcmd cannot attach, the service stops answering).

We saw it twice in 4 hours on 2026-10-07, on 2 hosts, with the same cycle each time. We had seen the same symptom (N cores at 100%, JVM unresponsive, no log line) every few hours on this service before we captured stacks, on x86_64 and on arm64 (Graviton) hosts. Both captures below are from x86_64 hosts.

Versions

  • dd-java-agent 1.65.1, which loads ddprof 1.49.0 (libjavaProfiler)
  • Profiling on (-Ddd.profiling.enabled=true), default profiler settings (CPU and wall-clock samplers)
  • Amazon Corretto 25.0.0+36 (JDK 25), x86_64, G1 GC
  • Amazon Linux 2023, glibc malloc, kernel 6.1
  • The application calls a native library (libvips, through glib) from many threads, so threads are often inside malloc.

JitCodeCache::DynamicCodeGenerated and JitCodeCache::findRuntimeStub in ddprof-lib/src/main/cpp/hotspot/jitCodeCache.cpp are the same in v_1.49.0, v_1.51.0 and main (checked on 2026-10-07), so we think the newer versions are affected too.

The cycle

Thread Holds Waits for
JVM Service Thread, posting a deferred DynamicCodeGenerated event JitCodeCache::_stubs_lock (exclusive, taken in DynamicCodeGenerated) the glibc malloc lock (malloc from CodeCache::add)
A worker thread interrupted by the profiler signal inside malloc (_int_malloc / malloc_consolidate) the glibc malloc lock _stubs_lock: the signal handler calls findRuntimeStub → SpinLock::lockShared(), which spins with no bound
Every other sampled thread — _stubs_lock, spinning in lockShared()

DynamicCodeGenerated calls _runtime_stubs.add() (which allocates) while it holds _stubs_lock, and findRuntimeStub takes the same lock from the signal handler with the unbounded lockShared(). Possible fixes: do not allocate under _stubs_lock (prepare the entry first), or use the bounded tryLockShared(max_spins) in findRuntimeStub when it runs in a signal handler and give up the frame.

Stacks, capture 1 (6 threads at 100% CPU, 5 of them in findRuntimeStub)

Collected with gdb -p <pid> -batch -ex "thread apply all bt". Addresses and paths removed.

Thread "Service Thread":
#0  __lll_lock_wait_private ()
#1  malloc ()
#2  CodeCache::add(void const*, int, char const*, bool) ()
#3  JitCodeCache::DynamicCodeGenerated(_jvmtiEnv*, char const*, void const*, int) ()
#4  JvmtiExport::post_dynamic_code_generated_internal(char const*, void const*, void const*) ()
#5  JvmtiDeferredEvent::post() ()
#6  ServiceThread::service_thread_entry(JavaThread*, JavaThread*) ()
#7  JavaThread::thread_main_inner() [clone .part.0] ()
#8  Thread::call_run() ()
#9  thread_native_entry(Thread*) ()
#10 start_thread ()
#11 clone3 ()

Thread "<HTTP worker thread>":
#0  JitCodeCache::findRuntimeStub(void const*) ()
#1  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, void const*, unsigned long, unsigned long, int, bool*) ()
#2  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, int, bool*) ()
#3  HotspotSupport::walkJavaStack(StackWalkRequest&) ()
#4  Profiler::recordSample(void*, unsigned long long, int, int, unsigned long long, Event*, unsigned long long*) ()
#5  CTimer::signalHandler(int, siginfo_t*, void*) ()
#6  <signal handler called> ()
#7  malloc_consolidate ()
#8  _int_malloc ()
#9  calloc ()
#10 g_malloc0 ()
#11 g_type_create_instance ()
#12 g_object_new_internal ()
#13 g_object_new_with_properties ()
   ...

Thread "<HTTP worker thread>":
#0  JitCodeCache::findRuntimeStub(void const*) ()
#1  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, void const*, unsigned long, unsigned long, int, bool*) ()
#2  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, int, bool*) ()
#3  HotspotSupport::walkJavaStack(StackWalkRequest&) ()
#4  Profiler::recordSample(void*, unsigned long long, int, int, unsigned long long, Event*, unsigned long long*) ()
#5  CTimer::signalHandler(int, siginfo_t*, void*) ()
#6  <signal handler called> ()
#7  hwy::x86::Cpuid (abcd=..., count=..., level=...)
#8  hwy::x86::FlagsFromCPUID ()
   ...

Stacks, capture 2 (8 threads at 100% CPU, all 8 in findRuntimeStub)

Thread "Service Thread":
#0  __lll_lock_wait_private ()
#1  malloc ()
#2  CodeCache::add(void const*, int, char const*, bool) ()
#3  JitCodeCache::DynamicCodeGenerated(_jvmtiEnv*, char const*, void const*, int) ()
#4  JvmtiExport::post_dynamic_code_generated_internal(char const*, void const*, void const*) ()
#5  JvmtiDeferredEvent::post() ()
#6  ServiceThread::service_thread_entry(JavaThread*, JavaThread*) ()
#7  JavaThread::thread_main_inner() [clone .part.0] ()
#8  Thread::call_run() ()
#9  thread_native_entry(Thread*) ()
#10 start_thread ()
#11 clone3 ()

Thread "<HTTP worker thread>":
#0  JitCodeCache::findRuntimeStub(void const*) ()
#1  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, void const*, unsigned long, unsigned long, int, bool*) ()
#2  HotspotSupport::walkVM(void*, _asgct_callframe*, int, StackWalkFeatures, EventType, int, bool*) ()
#3  HotspotSupport::walkJavaStack(StackWalkRequest&) ()
#4  Profiler::recordSample(void*, unsigned long long, int, int, unsigned long long, Event*, unsigned long long*) ()
#5  CTimer::signalHandler(int, siginfo_t*, void*) ()
#6  <signal handler called> ()
#7  _int_malloc ()
#8  calloc ()
#9  g_malloc0 ()
#10 g_closure_new_simple ()
#11 g_cclosure_new ()
#12 g_signal_connect_data ()
#13 ??? ()
   ...

top -H on both hosts showed the worker threads at 99.9% CPU each, while the VM Thread was sleeping (state S, under 4% CPU). jcmd <pid> Thread.print failed with AttachNotSupportedException ... doesn't respond within 10500ms.

We worked around it with DD_PROFILING_DDPROF_ENABLED=false (JFR-based profiling). We can test a fix build if useful.

Langage dominant
C++
Étoiles
33
Forks
14
Merge moyen
2 j 4 h
PR mergées (30 j)
52

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de DataDog/java-profiler

Toutes les issues de DataDog/java-profiler

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.