Reduce Solr JVM heap to leave room for OS page cache

未关闭
#265 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
15/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
停滞
技术栈
helm, python

调研方向

The issue says the server-side heap setting lives in the Helm chart, which is outside this repository's scope; it names no chart files or tests to inspect here. Confirm the deployment chart and its memory settings with the maintainers, then verify after deployment using /status?full=true and the listed heap, free-memory, and slow-query metrics.

由索引模型根据 Issue 内容生成。

描述

performance

Summary

Production /status is reporting a JVM heap that consumes nearly all of the host RAM, leaving the OS page cache too small to hold meaningful portions of the index. This is the most likely root cause of the bimodal latency we are seeing on the v1.5.2 instance:

  • p50 = 78 ms (warm path)
  • p95 = 4.3 s
  • p99 = 42 s
  • 12 % of queries classified as "very slow"

Evidence (from /status?full=true)

Metric Value
solr.os.total_physical_memory_bytes 34,359,738,368 (≈ 32 GiB)
solr.jvm.heap_max_bytes 32,212,254,720 (≈ 30 GiB)
solr.os.free_physical_memory_bytes 647,168 (≈ 632 KB)
solr.os.physical_memory_used_pct 1.0 (100 %)
solr.size 142.17 GB
solr.os.process_cpu_load / system_cpu_load 0

The heap is consuming ~94 % of host RAM. Lucene relies on the OS page cache for index segment access — with a 142 GB index against ~2 GiB of free RAM for filesystem cache, query tail latencies are dominated by disk paging. CPU load reading 0 while latency is high is consistent with I/O wait.

Suggested action (Helm chart)

  1. Drop the heap (-Xmx) to ~12–16 GiB.
  2. Increase node memory request/limit so at least 64 GiB (ideally more) is available, leaving ≥ 48 GiB for the OS page cache.

Solr/Lucene operations guidance recommends a small heap and a large filesystem cache for read-heavy workloads. The data-loading/Dockerfile currently sets -m 64G and data-loading/Makefile sets SOLR_MEM=220G — those are loader-side; the server-side setting lives in the Helm chart and is out of scope for this repo.

How to verify after deploy

After the change, hit /status?full=true and confirm:

  • solr.jvm.heap_max_bytes matches the new -Xmx
  • solr.os.free_physical_memory_bytes is in the tens of GiB, not hundreds of KiB
  • recent_queries.latency_buckets.very_slow_pct drops from 0.12 toward something close to 0
主要语言
Python
星标
5
派生
4
PR 合并指标
30 天内没有已合并 PR

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NCATSTranslator/NameResolution 的其他 Issue

查看 NCATSTranslator/NameResolution 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。