Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug] HStore Server image (`hugegraph/server`) keeps crash diagnostics only inside the container, so a restart on Kubernetes loses them

Đã đóng
  • Có gì không ổn?
#3,256 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@byteayan đang làm issue này rồi.

Từ ngày 2/10/2026.

  • #509 của @byteayan — đang mở
  • #3258 của @byteayan — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
52/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
docker, java, kubernetes, shell
Lĩnh vực
devops, infrastructure, observability

Hướng nghiên cứu

Start with Dockerfile-hstore and compare it with the standalone Dockerfile, then inspect log4j2.xml and hugegraph-server.sh at the referenced lines; docker-entrypoint.sh shows how JAVA_OPTS reaches the script. Check docker/README.md for the deployment documentation. Done means startup diagnostics reach Kubernetes stdout, WARN-level errors from the named loggers do too, JVM crash files use the logs directory, and that directory is documented for mounting.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Bug Type (问题类型)

server status (启动/运行异常)

Before submit
  • I have confirmed and searched that there are no similar problems in the historical issue and documents

Searched open and closed issues and PRs for hs_err, ErrorFile, HeapDumpPath, additivity, STDOUT_MODE, Dockerfile-hstore, docker logs, kubectl logs and Starting HugeGraphServer failed. #2979 / #2980 wired the console appender and set STDOUT_MODE=true in hugegraph-server/Dockerfile (the standalone hugegraph/hugegraph image), but not in hugegraph-server/Dockerfile-hstore, which builds hugegraph/server (docker/README.md:240). #3203 hit the same wall during a cold-start failure but tracked the retry bug, not the logging.

Environment (环境信息)
  • Server Version: hugegraph/server images. 2026-09-10: published latest (built 2026-09-08) and images built from master 60c8803d (#3203). 2026-09-22: images built from master 83ef9f3. All line numbers below are from master 176fb56dd (2026-10-02).
  • Backend: hstore, 3 PD + 3 Store + 3 Server
  • OS: Kubernetes (kind), HugeGraph Helm chart (#3218, earlier #3132)
  • Data Size: empty (2026-09-10) or a few test writes (2026-09-22); the failure is at Server startup
Expected & Actual behavior (期望与实际表现)

Observed. In the 2026-09-10 run (#3203), kubectl logs --previous for a Server that exited during startup ended with these lines and held no stack trace:

Connecting to HugeGraphServer (http://0.0.0.0:8080/graphs)...........Starting HugeGraphServer failed
See /hugegraph-server/logs/hugegraph-server.log for HugeGraphServer log output.

(printed by util.sh:362 and start-hugegraph.sh:117). The second line is the non-STDOUT_MODE branch. The cause was in logs/hugegraph-server.log, which went away with the container; the trace in #3203 was only recovered by mounting a volume at /hugegraph-server/logs before the first boot. On 2026-09-22 one Server started during a helm upgrade exited once with Starting HugeGraphServer failed, and its cause was lost the same way.

Expected. After a Server container exits, its stdout (kubectl logs --previous) shows the error, and the JVM crash files sit in a documented directory an operator can mount.

Actual. Four things keep diagnostics inside the container filesystem.

  1. The HStore image does not set STDOUT_MODE. Dockerfile-hstore sets only JAVA_OPTS and HUGEGRAPH_HOME (:43-44), and the published hugegraph/server:latest (amd64 config, created 2026-10-01, read from the Docker Hub registry on 2026-10-02) has no STDOUT_MODE in its Env. Without it, hugegraph-server.sh sends the JVM's stdout and stderr, console appender included, to logs/hugegraph-server-stdout.log (:266-270), so nothing the JVM logs reaches kubectl logs.

  2. Even with STDOUT_MODE=true, five loggers write only to the file. log4j2.xml sets LOG_PATH to logs (:21) and the file appender writes ${LOG_PATH}/hugegraph-server.log (:32). Since #2980 the root logger and org.apache.hugegraph also go to console (:106-109, :126-129). org.apache.hadoop, org.apache.zookeeper, com.alipay.sofa, io.netty and org.apache.commons are additivity="false" with only file (:110-124), so their errors never reach stdout. The file appender also has immediateFlush="false" (:34), so a killed JVM can lose its last lines from the file as well. The audit loggers (:130-135) and the slow-query log (:136-138) are file-only too. They are separate streams and out of scope here.

  3. JVM crash files land inside the install directory. hugegraph-server.sh sets LOGS="$TOP/logs" (:41), passes -XX:HeapDumpPath=${LOGS} (:124) and runs cd "${TOP}" before exec java (:91). Nothing in the repository sets -XX:ErrorFile (git grep ErrorFile on 176fb56dd returns nothing), so HotSpot writes hs_err_pid<pid>.log to the working directory, /hugegraph-server in the image. The image adds nothing here. JAVA_OPTS (Dockerfile-hstore:43) reaches the script through -j (docker-entrypoint.sh:243) and is appended on line 124. Line 124 sits inside the JAVA_OPTIONS default block (:118-129), so a user who sets JAVA_OPTIONS also loses -XX:+HeapDumpOnOutOfMemoryError.

  4. The only persistence is an image VOLUME. Dockerfile-hstore has WORKDIR /hugegraph-server/ (:46) and VOLUME /hugegraph-server (:78). Docker keeps that anonymous volume across docker restart. Kubernetes does not turn an image VOLUME into a pod volume. A restart starts a new container, and logs/, hs_err_pid*.log and heap dumps go with the old one.

Fix direction.

  • Set STDOUT_MODE="true" in Dockerfile-hstore, as #2980 did in Dockerfile.
  • In log4j2.xml, also send WARN and above from the five loggers on lines 110-124 to console (a console ref with a WARN threshold, or a second console appender). INFO from netty and hadoop stays in the file. Audit and slow-query stay file-only.
  • In hugegraph-server.sh, add -XX:ErrorFile=${LOGS}/hs_err_pid%p.log and move -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} out of the default block, so both apply whatever JAVA_OPTIONS holds.
  • Document /hugegraph-server/logs as the directory to mount on Kubernetes. If #3253 (for #3236) lands, these flags should follow its LOGS_OVERRIDE instead of $TOP/logs.

Related: #2979 / #2980, #3203, #3236 / #3253, #3211 (non-root image, which would chown the same log directory).

Vertex/Edge example (问题点 / 边数据举例)

Not applicable.

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

Not applicable.

Ngôn ngữ chính
Java
Star
3.2k
Fork
641
Merge trung bình
1 ngày 9 giờ
Pull request đã merge (30 ngày)
60

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/hugegraph

Tất cả issue của apache/hugegraph

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.