Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug] HStore Server image (`hugegraph/server`) keeps crash diagnostics only inside the container, so a restart on Kubernetes loses them

Aperta
  • Qualcosa non va?
#3,256 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@byteayan ci sta già lavorando.

Dal 2/10/2026.

  • #509 di @byteayan — aperta
  • #3258 di @byteayan — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
docker, java, kubernetes, shell
Ambito
devops, infrastructure, observability

Direzione di ricerca

Start with Dockerfile-hstore and compare it with the standalone Dockerfile, then inspect log4j2.xml and hugegraph-server.sh at the referenced lines; docker-entrypoint.sh shows how JAVA_OPTS reaches the script. Check docker/README.md for the deployment documentation. Done means startup diagnostics reach Kubernetes stdout, WARN-level errors from the named loggers do too, JVM crash files use the logs directory, and that directory is documented for mounting.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug Type (问题类型)

server status (启动/运行异常)

Before submit
  • I have confirmed and searched that there are no similar problems in the historical issue and documents

Searched open and closed issues and PRs for hs_err, ErrorFile, HeapDumpPath, additivity, STDOUT_MODE, Dockerfile-hstore, docker logs, kubectl logs and Starting HugeGraphServer failed. #2979 / #2980 wired the console appender and set STDOUT_MODE=true in hugegraph-server/Dockerfile (the standalone hugegraph/hugegraph image), but not in hugegraph-server/Dockerfile-hstore, which builds hugegraph/server (docker/README.md:240). #3203 hit the same wall during a cold-start failure but tracked the retry bug, not the logging.

Environment (环境信息)
  • Server Version: hugegraph/server images. 2026-09-10: published latest (built 2026-09-08) and images built from master 60c8803d (#3203). 2026-09-22: images built from master 83ef9f3. All line numbers below are from master 176fb56dd (2026-10-02).
  • Backend: hstore, 3 PD + 3 Store + 3 Server
  • OS: Kubernetes (kind), HugeGraph Helm chart (#3218, earlier #3132)
  • Data Size: empty (2026-09-10) or a few test writes (2026-09-22); the failure is at Server startup
Expected & Actual behavior (期望与实际表现)

Observed. In the 2026-09-10 run (#3203), kubectl logs --previous for a Server that exited during startup ended with these lines and held no stack trace:

Connecting to HugeGraphServer (http://0.0.0.0:8080/graphs)...........Starting HugeGraphServer failed
See /hugegraph-server/logs/hugegraph-server.log for HugeGraphServer log output.

(printed by util.sh:362 and start-hugegraph.sh:117). The second line is the non-STDOUT_MODE branch. The cause was in logs/hugegraph-server.log, which went away with the container; the trace in #3203 was only recovered by mounting a volume at /hugegraph-server/logs before the first boot. On 2026-09-22 one Server started during a helm upgrade exited once with Starting HugeGraphServer failed, and its cause was lost the same way.

Expected. After a Server container exits, its stdout (kubectl logs --previous) shows the error, and the JVM crash files sit in a documented directory an operator can mount.

Actual. Four things keep diagnostics inside the container filesystem.

  1. The HStore image does not set STDOUT_MODE. Dockerfile-hstore sets only JAVA_OPTS and HUGEGRAPH_HOME (:43-44), and the published hugegraph/server:latest (amd64 config, created 2026-10-01, read from the Docker Hub registry on 2026-10-02) has no STDOUT_MODE in its Env. Without it, hugegraph-server.sh sends the JVM's stdout and stderr, console appender included, to logs/hugegraph-server-stdout.log (:266-270), so nothing the JVM logs reaches kubectl logs.

  2. Even with STDOUT_MODE=true, five loggers write only to the file. log4j2.xml sets LOG_PATH to logs (:21) and the file appender writes ${LOG_PATH}/hugegraph-server.log (:32). Since #2980 the root logger and org.apache.hugegraph also go to console (:106-109, :126-129). org.apache.hadoop, org.apache.zookeeper, com.alipay.sofa, io.netty and org.apache.commons are additivity="false" with only file (:110-124), so their errors never reach stdout. The file appender also has immediateFlush="false" (:34), so a killed JVM can lose its last lines from the file as well. The audit loggers (:130-135) and the slow-query log (:136-138) are file-only too. They are separate streams and out of scope here.

  3. JVM crash files land inside the install directory. hugegraph-server.sh sets LOGS="$TOP/logs" (:41), passes -XX:HeapDumpPath=${LOGS} (:124) and runs cd "${TOP}" before exec java (:91). Nothing in the repository sets -XX:ErrorFile (git grep ErrorFile on 176fb56dd returns nothing), so HotSpot writes hs_err_pid<pid>.log to the working directory, /hugegraph-server in the image. The image adds nothing here. JAVA_OPTS (Dockerfile-hstore:43) reaches the script through -j (docker-entrypoint.sh:243) and is appended on line 124. Line 124 sits inside the JAVA_OPTIONS default block (:118-129), so a user who sets JAVA_OPTIONS also loses -XX:+HeapDumpOnOutOfMemoryError.

  4. The only persistence is an image VOLUME. Dockerfile-hstore has WORKDIR /hugegraph-server/ (:46) and VOLUME /hugegraph-server (:78). Docker keeps that anonymous volume across docker restart. Kubernetes does not turn an image VOLUME into a pod volume. A restart starts a new container, and logs/, hs_err_pid*.log and heap dumps go with the old one.

Fix direction.

  • Set STDOUT_MODE="true" in Dockerfile-hstore, as #2980 did in Dockerfile.
  • In log4j2.xml, also send WARN and above from the five loggers on lines 110-124 to console (a console ref with a WARN threshold, or a second console appender). INFO from netty and hadoop stays in the file. Audit and slow-query stay file-only.
  • In hugegraph-server.sh, add -XX:ErrorFile=${LOGS}/hs_err_pid%p.log and move -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} out of the default block, so both apply whatever JAVA_OPTIONS holds.
  • Document /hugegraph-server/logs as the directory to mount on Kubernetes. If #3253 (for #3236) lands, these flags should follow its LOGS_OVERRIDE instead of $TOP/logs.

Related: #2979 / #2980, #3203, #3236 / #3253, #3211 (non-root image, which would chown the same log directory).

Vertex/Edge example (问题点 / 边数据举例)

Not applicable.

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

Not applicable.

Lingua principale
Java
Stelle
3.2k
Fork
641
Merge medio
2g 9h
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/hugegraph

Tutte le issue di apache/hugegraph

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.