Metaspace leak in Spring Boot / Tomcat apps after upgrading to 1.64.0 — tied to DD_APPSEC_SCA_ENABLED

Tracer Version(s)

1.64.0

Java Version(s)

21.0.11

JVM Vendor

Amazon Corretto

Bug Report

After upgrading dd-trace-java from 1.63.2 to 1.64.0, our Spring Boot applications (Tomcat based) started running out of Metaspace within about a day of deployment. Metaspace grows continuously in a straight line rather than plateauing, until it hits the configured limit and the JVM/pod is killed.

Downgrading to 1.63.2 with no other changes resolves the issue. On 1.64.0, setting DD_APPSEC_SCA_ENABLED=false also resolves it (all other flags unchanged). This strongly points at the SCA reachability work shipped in 1.64.0 (possibly #11352 "Implement SCA Reachability runtime detection" and/or #11614 "Migrate SCA Reachability to method-level symbol database") as the likely root cause, but we haven't been able to confirm the exact mechanism from our side.

I created a Datadog Helpdesk issue for this as well at Request #2955022 (with uploaded class load file).

Expected Behavior

Expected behavior

Metaspace usage should stabilize after the application warms up (JIT/class loading settles), as it did on 1.63.2 and as it does on 1.64.0 when Datadog instrumentation is fully disabled.

Actual behavior

Metaspace (jvm.gc.metaspace_size) grows continuously and roughly linearly for the life of the process. Example from one affected pod:

15:00 — Metaspace: ~110 MB
06:00 next day — Metaspace: ~200 MB

Growth does not plateau; left running, the process eventually hits the configured Metaspace limit and OOMs. This started the same day we rolled out a new image built against 1.64.0 — no other code or config changes shipped alongside it.

Reproduction Code

Run a Spring Boot (Tomcat) service on Java 21 with dd-java-agent 1.64.0 and the configuration above (in particular DD_APPSEC_SCA_ENABLED=true).
Let it run under normal traffic for several hours while monitoring jvm.gc.metaspace_size (or jcmd VM.metaspace / a JFR/heap capture).
Observe continuous, non-plateauing Metaspace growth until the container is OOM-killed (roughly within 24h in our environment, depending on Metaspace limit and traffic).