SIGSEGV crash in Dictionary::lookup when profiling with Pyroscope

Open
#368 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
java, kubernetes, linux

Research direction

Start at the native Dictionary::lookup entry point named in the crash and review the async-profiler path used when CPU, ALLOC, and LOCK profiling run together. Reproduce the configuration in the reported Linux container environment and compare the fatal error details across affected instances. Done means identifying and preventing the SIGSEGV in this profiling combination.

Written by the indexing model from the issue text.

Description

Hello:

We are experiencing intermittent JVM crashes (SIGSEGV) on a small number of container instances when using async-profiler via Pyroscope. The crash occurs in the native Dictionary::lookup function.
Most containers run fine — only a few pods are affected, which suggests a possible race condition or memory corruption issue.

How Pyroscope is started:
We enable CPU, ALLOC, and LOCK profiling simultaneously via the Pyroscope Java agent:
java -javaagent:pyroscope.jar
-Dpyroscope.application.name=***
-Dpyroscope.server.address=***
-Dpyroscope.profiler.event=CPU
-Dpyroscope.profiler.alloc=4096k
-Dpyroscope.profiler.lock=20ms
-jar app.jar

Environment:
OS: Linux (amd64), containerized (Kubernetes)

  • JRE: Java(TM) SE Runtime Environment 17.0.10+11-LTS-240
  • JVM: Java HotSpot(TM) 64-Bit Server VM (mixed mode, sharing, tiered, compressed oops, compressed class ptrs, G1 GC)
  • Pyroscope version: 2.9.1
  • Container base image: <e.g., oracle/graalvm-ce:17, eclipse-temurin:17, etc.>

Crash details:
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007f3bf4e08965, pid=1, tid=596
JRE version: Java(TM) SE Runtime Environment (17.0.10+11) (build 17.0.10+11-LTS-240)
Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0.10+11-LTS-240, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
Problematic frame:
C [libasyncProfiler-linux-x64-94af16d95b27e7e3346a51e8d43e744de92a62a3.so+0xd965] Dictionary::lookup(char const*, unsigned long)+0x65
An error report file with more information is saved as:

If you would like to submit a bug report, please visit:
https://bugreport.java.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f44abbdabc7]
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f44abbdabc7]
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f44abbdabc7]
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f44abbdabc7]
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f44abbdabc7]
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f44abbdabc7]

Dominant language
Java
Stars
125
Forks
48
Avg merge
1h 8m
Merged PRs (30d)
1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from grafana/pyroscope-java

All issues in grafana/pyroscope-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.