[Bug] IotDB pods crash with OOM because we calculate the memory based on the node not pod resources
还没有人认领这个 Issue。
评估
调研方向
从 conf/confignode-env.sh 和 conf/datanode-env.sh 开始,重点检查使用 free -m 进行的 system_memory_in_mb 计算。比较 issue 中描述的 cgroup v2 和 v1 内存限制情况,然后验证限制为 8 GB 的 pod 是否根据该限制而不是主机内存来确定 JVM 大小,并避免因 OOM 重启。
由索引模型根据 Issue 内容生成。
描述
Search before asking
- I searched in the issues and found nothing similar.
Version
latest
Describe the bug and provide the minimal reproduce step
Start IOTDB datanode and confignode pods with memory limits, for example 8 GB, and allocate 8 GB of resources to each pod.
Pods keep crashing with OOM errors because the JVM is trying to allocate 16 GB of memory.
What did you expect to see?
# When running in a container/pod, use cgroup memory limit instead of host memory
if [ -f /sys/fs/cgroup/memory.max ]; then
# cgroup v2
cgroup_mem=`cat /sys/fs/cgroup/memory.max`
if [ "$cgroup_mem" != "max" ]; then
cgroup_mem_in_mb=`expr $cgroup_mem / 1024 / 1024`
if [ "$cgroup_mem_in_mb" -lt "$system_memory_in_mb" ]; then
system_memory_in_mb=$cgroup_mem_in_mb
fi
fi
elif [ -f /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
# cgroup v1
cgroup_mem=`cat /sys/fs/cgroup/memory/memory.limit_in_bytes`
cgroup_mem_in_mb=`expr $cgroup_mem / 1024 / 1024`
if [ "$cgroup_mem_in_mb" -lt "$system_memory_in_mb" ]; then
system_memory_in_mb=$cgroup_mem_in_mb
fi
fi
8GB
I would expect the memory to be auto-calculated based on the pod resources (8 GB), not the node resources (32 GB).
# scripts\conf\datanode-env.sh
system_memory_in_mb=`free -m | sed -n '2p' | awk '{print \$2}'` returns 32 GB.
What did you see instead?
32 GB and a lot of pod restarts
Anything else?
# iotdb\WORKING_CONFIGS.md
## 2) JVM Memory (Linux)
Edit these files:
- conf/confignode-env.sh
- conf/datanode-env.sh
Set MEMORY_SIZE explicitly to avoid auto-sizing surprises.
### ConfigNode memory
```bash
# conf/confignode-env.sh
MEMORY_SIZE=2G
DataNode memory
# conf/datanode-env.sh
MEMORY_SIZE=8G
Why are there no env varibales for this setting?
Do you expect the clouad env to manualy go and change this limit?
This is a hack, and we should not have to do this in a pod
- IOTDB_JMX_OPTS=-Xmx4G
Are you willing to submit a PR?
- I'm willing to submit a PR!
- 主要语言
- Java
- 星标
- 6.4k
- 派生
- 1.2k
- 平均合并
- 1 天 17 小时
- 30 天内合并 PR
- 152
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/iotdb 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 82/100
-
IoTDB Edge: stop-edge.sh does not stop its own process when IOTDB_HOME is set, and reports success 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
-
难度 2/5 1-3 小时 新手友好度 78/100
-
难度 2/5 1-3 小时 新手友好度 78/100
-
难度 2/5 1-3 小时 新手友好度 78/100
相似的 Issue
-
documentation
难度 2/5 1-3 小时 新手友好度 65/100
inu-appcenter/memorIN-backend#288 ·
-
难度 2/5 1-3 小时 新手友好度 65/100
-
frontend maui-pilot pilot-ask question
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 75/100
-
area/plugin
难度 2/5 1-3 小时 新手友好度 75/100
kestra-io/plugin-kestra#190 ·