[bug] Issues mounting extended disk in ppc64

Abierto
#894 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 4 días

@deekay2310 ya está trabajando en esto.

Desde el 14/9/2026.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
docker, gitlab, shell

Línea de trabajo

Comienza con install-glrunner.sh y localiza el generador de executor-containers.conf y la sección [runners.docker] de config.toml. Revisa cómo se seleccionan los valores de DNS específicos de la región y, después, valida los cambios de subred, DNS, modo de red y MTU con trabajos de podman anidados ejecutándose simultáneamente en los runners ppc64le; se considera terminado cuando esos trabajos se completan sin fallos de DNS ni de enrutamiento.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

mapt issue: ibm-power: fix podman networking for nested containers in GitLab runner builds

Background

The install-glrunner.sh script configures podman networking for the IBM Power (ppc64le)
GitLab runners. Several configuration bugs cause DNS failures inside build containers,
particularly in jobs that run nested podman run calls from within the gitlab-runner build
container (e.g. bootstrap_and_onboard.sh pattern).


Issues found

1. executor-containers.conf — wrong default_subnet causes routing conflict

File: whatever generates executor-containers.conf (mounted into build containers as
/etc/containers/containers.conf:ro)

The file currently sets:

[network]
default_subnet = "192.168.100.0/24"

With network_mode = "podman" in config.toml (see issue 3 below), all gitlab-runner build
containers are attached to the host's podman network at 192.168.100.0/24. When a build
container's job script runs a nested podman run, podman inside the build container reads
this containers.conf and tries to create a bridge at 192.168.100.1/24 inside the build
container's network namespace — the same subnet as the build container's own eth0.
This creates a conflicting route inside the build container, breaking all outbound traffic
from nested containers including DNS.

Fix: use a subnet that does not conflict with the host's podman network:

[network]
default_subnet = "172.20.0.0/24"

2. executor-containers.conf — no public DNS fallback

The file only lists IT-internal DNS resolvers:

dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69"]
dns_options = ["timeout:2", "attempts:5", "single-request"]

If the internal resolvers are temporarily unreachable (VPN/TGW hiccup), DNS fails
completely in nested containers with no fallback. attempts:5 also makes the worst-case
timeout 30 seconds before failure.

Fix:

dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
dns_options = ["timeout:2", "attempts:3", "single-request"]

3. config.toml — missing network_mode causes concurrent bridge race condition

With concurrent = 2, gitlab-runner creates two per-job bridge networks simultaneously.
The second bridge's nftables masquerade rules may not be applied before its containers
start sending traffic, causing intermittent DNS timeouts under concurrent load.

Fix: add to the [runners.docker] section:

network_mode = "podman"

This makes all job containers share the existing host podman network instead of creating
per-job bridges, eliminating the race condition entirely.


4. config.toml — missing explicit DNS for build containers

Without dns = in config.toml, gitlab-runner passes an empty DNS list via the Docker API,
which may override the host's containers.conf dns_servers setting for build containers.

Fix: add to [runners.docker]:

dns = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]

5. config.toml — missing MTU setting

PowerVS network interfaces have an MTU of 1450 (lower than the default 1500). Without
setting this, large packets (e.g. during image pulls) may be silently dropped.

Fix: add to [runners.docker]:

network_mtu = 1450

Required final configuration

executor-containers.conf
[containers]
log_driver = "journald"
pids_limit = -1
dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
dns_options = ["timeout:2", "attempts:3", "single-request"]

[network]
default_subnet = "172.20.0.0/24"
config.toml — [runners.docker] section
[runners.docker]
  host = "unix:///run/podman/podman.sock"
  privileged = true
  network_mode = "podman"
  network_mtu = 1450
  dns = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
  volumes = ["/etc/containers/executor-containers.conf:/etc/containers/containers.conf:ro", "/cache"]

Notes

  • The IT DNS resolvers (10.130.98.x) above are for us-south. Other regions use
    different addresses — the install script should parameterise these per region:

    Region DNS resolvers
    us-east 10.130.64.4, 10.130.64.70, 10.130.64.134
    us-south 10.130.98.5, 10.130.98.69, 10.130.98.132
    eu-de 10.130.128.5, 10.130.128.69, 10.130.128.133
    lon 10.130.129.4, 10.130.129.68, 10.130.129.132
  • The 172.20.0.0/24 subnet for executor-containers.conf must not conflict with the
    host's podman network (192.168.100.0/24), the PowerVS subnet, or the 10.0.0.0/8
    internal route.

  • All fixes were validated on live dal14 ppc64le runners (aipcc-pro-gl-ppc64le-1..4).
    Post-fix jobs ran successfully including nested podman run workloads
    (bootstrap-and-onboard jobs across all 4 runners, 2 confirmed successe

Lenguaje dominante
Go
Estrellas
25
Forks
41
Merge medio
5 d 3 h
PR fusionados (30 d)
16

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de redhat-developer/mapt

Todos los issues de redhat-developer/mapt

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.