mgmt-hub: OpenBao auth plugin is staged in /tmp, so systemd-tmpfiles deletes it and the bao container fails to start (exit 127)

Abierto
  • ¿Algo no está bien?
#231 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
66/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
docker, docker-compose, shell
Área
devops, infrastructure

Línea de trabajo

Comienza con mgmt-hub/deploy-mgmt-hub.sh alrededor de la línea 290 y las líneas 770-771; después, inspecciona el bind mount del plugin en mgmt-hub/docker-compose.yml alrededor de la línea 244. Reproduce el fallo eliminando /tmp/horizon-all-in-1 y recreando el contenedor bao. Se considera terminado cuando el plugin sobrevive a la limpieza de tmpfiles y bao se inicia correctamente después de la recreación.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

Summary

The all-in-one management hub stages the OpenBao auth plugin binary under /tmp and bind-mounts it into the bao container directly from there:

/tmp is not durable storage. On a stock Ubuntu host, /usr/lib/tmpfiles.d/tmp.conf contains:

D /tmp 1777 root root 30d

which means systemd-tmpfiles both wipes /tmp at every boot and deletes files older than 30 days during the daily systemd-tmpfiles-clean.timer run. The plugin binary is written once at install time and never touched again, so it reliably ages out after 30 days.

Why it fails silently, then breaks all at once

The running bao container holds the bind mount by inode, so it keeps working normally after the host file is deleted. Nothing is logged and nothing appears wrong — until the next time the container is recreated (Docker daemon upgrade/restart, host reboot, docker-compose up). At that point the mount source no longer exists.

Docker then auto-creates the missing bind source as a directory, while the container-side target is a file, producing a confusing error that does not mention the real problem:

failed to create task for container: failed to create shim task: OCI runtime create failed:
runc create failed: unable to start container process: error during container init:
error mounting "/tmp/horizon-all-in-1/openbao/plugins/openbao-plugin-auth-openhorizon" to rootfs at
"/openbao/plugins/openbao-plugin-auth-openhorizon": ... not a directory:
Are you trying to mount a directory onto a file (or vice-versa)?

The container exits 127. Note that although the service is declared restart: always, this is a create-time failure, so RestartCount stays at 0 and Docker never retries — the hub simply comes up permanently without OpenBao.

In our case the hub was installed 2026-05-23 and ran fine for two months. systemd-tmpfiles-clean.service last ran 2026-07-22, and a Docker daemon restart the following day left bao dead with the error above.

Steps to reproduce

  1. Install the all-in-one management hub on a stock Ubuntu host:

    curl -sSL https://raw.githubusercontent.com/open-horizon/devops/master/mgmt-hub/deploy-mgmt-hub.sh -o deploy-mgmt-hub.sh
chmod +x deploy-mgmt-hub.sh && ./deploy-mgmt-hub.sh

  2. Confirm the plugin is staged in /tmp and the container is healthy:

    ls -l /tmp/horizon-all-in-1/openbao/plugins/openbao-plugin-auth-openhorizon
docker ps --filter name=bao

  3. Simulate the tmpfiles cleanup that occurs after 30 days (or a reboot):

    sudo rm -rf /tmp/horizon-all-in-1

    The bao container is still running and healthy at this point — this is the silent window.

  4. Restart the container the way a Docker upgrade or reboot would:

    docker restart bao
docker ps -a --filter name=bao      # Exited (127)
docker inspect bao --format '{{.State.Error}}'

To observe it without waiting 30 days, you can instead confirm the aging rule directly with grep '^[dD] /tmp' /usr/lib/tmpfiles.d/tmp.conf and check systemctl status systemd-tmpfiles-clean.timer.

Suggested improvements

In rough order of preference:

  1. Stage the plugin somewhere persistent. Anywhere outside /tmp fixes the root cause — e.g. a openbao/plugins/ directory next to the generated docker-compose.yml in the install directory, or /var/lib/horizon/openbao/plugins. This is a one-line change to TMP_DIR usage for this specific artifact plus the compose mount.

  2. Or hold the plugin in a named Docker volume (e.g. bao-plugins-vol), populated at install time and mounted at /openbao/plugins. This puts the plugin under the same lifecycle management as bao-vol and bao-logs-vol, which are already named volumes, and keeps it consistent with how the rest of OpenBao's state is handled.

  3. Mount the plugin directory rather than the individual file. Bind-mounting a single file is what turns a missing source into the misleading "directory onto a file" error. Mounting .../openbao/plugins → /openbao/plugins fails more clearly and also allows the plugin to be replaced/upgraded without editing the compose file.

  4. Add a preflight check. Have deploy-mgmt-hub.sh verify the plugin binary exists before bringing up bao and re-stage it if not — ideally comparing against the SHA256 already registered in the OpenBao plugin catalog, since a mismatch there is its own failure mode.

  5. If /tmp must be retained, ship a tmpfiles.d drop-in that exempts the path from cleanup and restores it at boot.

Workaround

For anyone hitting this now: re-download the plugin matching the version in OPENBAO_PLUGIN_AUTH_OPENHORIZON_VERSION (checksums are published on the plugin's release page), place it somewhere persistent, and restore it. The registered SHA256 in the catalog must match the binary, so the same release version must be used. The container then starts, and OpenBao needs unsealing again with the key from the install summary.

We additionally added an /etc/tmpfiles.d/ drop-in so the file is re-copied into /tmp at boot and exempted from age-based cleaning, which survives reboots without modifying the upstream compose file:

d /tmp/horizon-all-in-1/openbao/plugins 0755 root root -
C /tmp/horizon-all-in-1/openbao/plugins/openbao-plugin-auth-openhorizon 0755 root root - /path/to/persistent/openbao-plugin-auth-openhorizon
x /tmp/horizon-all-in-1

Environment
Deployment deploy-mgmt-hub.sh all-in-one, defaults
OpenBao 2.0.3 (quay.io/openbao/openbao-ubi:2.0)
Plugin openbao-plugin-auth-openhorizon v1.0.1
Host Ubuntu 24.04 (kernel 6.8), Docker 29.6.2
Lenguaje dominante
Shell
Estrellas
12
Forks
46
Merge medio
5 d 12 h
PR fusionados (30 d)
3

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de open-horizon/devops

Todos los issues de open-horizon/devops

Issues similares

Más issues de Shell/Bash

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.