Reduce Solr JVM heap to leave room for OS page cache

Abierto
#265 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
15/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
helm, python

Línea de trabajo

The issue says the server-side heap setting lives in the Helm chart, which is outside this repository's scope; it names no chart files or tests to inspect here. Confirm the deployment chart and its memory settings with the maintainers, then verify after deployment using /status?full=true and the listed heap, free-memory, and slow-query metrics.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

performance

Summary

Production /status is reporting a JVM heap that consumes nearly all of the host RAM, leaving the OS page cache too small to hold meaningful portions of the index. This is the most likely root cause of the bimodal latency we are seeing on the v1.5.2 instance:

  • p50 = 78 ms (warm path)
  • p95 = 4.3 s
  • p99 = 42 s
  • 12 % of queries classified as "very slow"

Evidence (from /status?full=true)

Metric Value
solr.os.total_physical_memory_bytes 34,359,738,368 (≈ 32 GiB)
solr.jvm.heap_max_bytes 32,212,254,720 (≈ 30 GiB)
solr.os.free_physical_memory_bytes 647,168 (≈ 632 KB)
solr.os.physical_memory_used_pct 1.0 (100 %)
solr.size 142.17 GB
solr.os.process_cpu_load / system_cpu_load 0

The heap is consuming ~94 % of host RAM. Lucene relies on the OS page cache for index segment access — with a 142 GB index against ~2 GiB of free RAM for filesystem cache, query tail latencies are dominated by disk paging. CPU load reading 0 while latency is high is consistent with I/O wait.

Suggested action (Helm chart)

  1. Drop the heap (-Xmx) to ~12–16 GiB.
  2. Increase node memory request/limit so at least 64 GiB (ideally more) is available, leaving ≥ 48 GiB for the OS page cache.

Solr/Lucene operations guidance recommends a small heap and a large filesystem cache for read-heavy workloads. The data-loading/Dockerfile currently sets -m 64G and data-loading/Makefile sets SOLR_MEM=220G — those are loader-side; the server-side setting lives in the Helm chart and is out of scope for this repo.

How to verify after deploy

After the change, hit /status?full=true and confirm:

  • solr.jvm.heap_max_bytes matches the new -Xmx
  • solr.os.free_physical_memory_bytes is in the tens of GiB, not hundreds of KiB
  • recent_queries.latency_buckets.very_slow_pct drops from 0.12 toward something close to 0
Lenguaje dominante
Python
Estrellas
5
Forks
4
Métricas de merge de PR
Sin PR fusionados en 30 d

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NCATSTranslator/NameResolution

Todos los issues de NCATSTranslator/NameResolution

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.