Published Dockerfiles set ENV NODE_HOSTNAME=localhost, silently disabling replication after an in-place 4.x→5.x upgrade

Start with the ENV NODE_HOSTNAME=localhost lines in Dockerfile, Dockerfile-gpu, and Dockerfile-openshift, then read getThisNodeName() in server/nodeName.ts to confirm the precedence described in the issue. Build or inspect the published image environment and verify that a clustered upgrade can use replication.hostname without the image supplying a localhost override.

Summary

All three published Dockerfiles set ENV NODE_HOSTNAME=localhost. Because Harper reads environment variables as configuration, that env var becomes node.hostname — which getThisNodeName() treats as the highest-priority source of node identity. On an in-place 4.x → 5.x upgrade of a clustered node, the node therefore identifies as localhost, does not match its row in system.hdb_nodes, and user-database replication silently stops running for that node.

The operator-facing symptom is worse than the misconfiguration itself: Harper writes the env-derived value back into harperdb-config.yaml on every boot, so editing or deleting node.hostname in the config file appears to "work" and then reverts on the next restart. There is no obvious signal that an environment variable is the thing overriding it.

Where

• Dockerfile:114 — ENV NODE_HOSTNAME=localhost
• Dockerfile-gpu:115 — same
• Dockerfile-openshift:118 — same

Identity resolution that consumes it (core): server/nodeName.ts getThisNodeName() checks node.hostname first and only falls back to replication.hostname, replication.url, the TLS CN, and finally the listening port.

Reproduction

On a two-node 4.x cluster whose nodes are registered in system.hdb_nodes as node-1.example.com / node-2.example.com, with replication.hostname correctly set per node:

1. Upgrade one node in place to 5.2.x using the published image.
2. On boot, the log reports:

Replication identity mismatch: this node identifies as "localhost" (url wss://localhost:9933)
but no matching row exists in system.hdb_nodes (which has 2 node(s): node-1.example.com,
node-2.example.com). User-database replication will NOT run for this node until its identity
matches a registered node name.

3. grep hostname harperdb-config.yaml shows both replication.hostname: node-1.example.com (correct) and node.hostname: localhost (wrong, and it wins).
4. Remove node.hostname from harperdb-config.yaml, or set it to node-1.example.com, and restart → the file is rewritten with localhost again. Config file mtime lands ~1s after container start.
5. docker inspect <container> --format '{{json .Config.Env}}' shows NODE_HOSTNAME=localhost, and docker history <image> attributes it to an ENV NODE_HOSTNAME=localhost layer, i.e. it comes from the image rather than the deployment.

Verified against harperfast/harper-pro:5.2.3.

Impact

• Any in-place upgrade of a clustered node to 5.x loses user-database replication for that node until an operator discovers the env var. Replication failing silently while the node otherwise serves traffic is the dangerous part — on an active-active pair it presents as data divergence rather than as an upgrade error.
• Every node from these images is affected identically, so an operator fixing node 1 will hit it again on node 2.
• node.hostname did not carry this meaning in 4.x, so the env var was inert there. Nothing in an upgrade runbook would flag it.

Suggested fix

Drop ENV NODE_HOSTNAME=localhost from all three Dockerfiles. If a default is wanted for single-node/dev container use, it should not be one that outranks replication.hostname on an upgraded cluster member — leaving it unset lets getThisNodeName() fall through to replication.hostname, which is already correct on a registered node.

Two smaller things that would each have saved most of the debugging time:

1. The mismatch warning is good, and correctly does not tell the operator to cement the wrong value. It could go one step further and name where the losing value came from (env var vs config file), since the config file is the misleading place to look.
2. Persisting env-derived config back into harperdb-config.yaml makes an env override indistinguishable from a file setting after the first boot. Worth considering whether env-sourced values should be persisted at all.