[bug] Issues mounting extended disk in ppc64

未关闭
#894 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 4 天内回复

@deekay2310 已经在做这个了。

开始于 2026年9月14日。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
45/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
docker, gitlab, shell

调研方向

从 install-glrunner.sh 开始,找到 executor-containers.conf 的生成器以及 config.toml 中的 [runners.docker] 部分。检查如何选择特定区域的 DNS 值,然后在 ppc64le runner 上并发运行嵌套的 podman 作业,以验证子网、DNS、网络模式和 MTU 的更改;这些作业在没有 DNS 或路由故障的情况下完成即表示完成。

由索引模型根据 Issue 内容生成。

描述

mapt issue: ibm-power: fix podman networking for nested containers in GitLab runner builds

Background

The install-glrunner.sh script configures podman networking for the IBM Power (ppc64le)
GitLab runners. Several configuration bugs cause DNS failures inside build containers,
particularly in jobs that run nested podman run calls from within the gitlab-runner build
container (e.g. bootstrap_and_onboard.sh pattern).


Issues found

1. executor-containers.conf — wrong default_subnet causes routing conflict

File: whatever generates executor-containers.conf (mounted into build containers as
/etc/containers/containers.conf:ro)

The file currently sets:

[network]
default_subnet = "192.168.100.0/24"

With network_mode = "podman" in config.toml (see issue 3 below), all gitlab-runner build
containers are attached to the host's podman network at 192.168.100.0/24. When a build
container's job script runs a nested podman run, podman inside the build container reads
this containers.conf and tries to create a bridge at 192.168.100.1/24 inside the build
container's network namespace — the same subnet as the build container's own eth0.
This creates a conflicting route inside the build container, breaking all outbound traffic
from nested containers including DNS.

Fix: use a subnet that does not conflict with the host's podman network:

[network]
default_subnet = "172.20.0.0/24"

2. executor-containers.conf — no public DNS fallback

The file only lists IT-internal DNS resolvers:

dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69"]
dns_options = ["timeout:2", "attempts:5", "single-request"]

If the internal resolvers are temporarily unreachable (VPN/TGW hiccup), DNS fails
completely in nested containers with no fallback. attempts:5 also makes the worst-case
timeout 30 seconds before failure.

Fix:

dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
dns_options = ["timeout:2", "attempts:3", "single-request"]

3. config.toml — missing network_mode causes concurrent bridge race condition

With concurrent = 2, gitlab-runner creates two per-job bridge networks simultaneously.
The second bridge's nftables masquerade rules may not be applied before its containers
start sending traffic, causing intermittent DNS timeouts under concurrent load.

Fix: add to the [runners.docker] section:

network_mode = "podman"

This makes all job containers share the existing host podman network instead of creating
per-job bridges, eliminating the race condition entirely.


4. config.toml — missing explicit DNS for build containers

Without dns = in config.toml, gitlab-runner passes an empty DNS list via the Docker API,
which may override the host's containers.conf dns_servers setting for build containers.

Fix: add to [runners.docker]:

dns = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]

5. config.toml — missing MTU setting

PowerVS network interfaces have an MTU of 1450 (lower than the default 1500). Without
setting this, large packets (e.g. during image pulls) may be silently dropped.

Fix: add to [runners.docker]:

network_mtu = 1450

Required final configuration

executor-containers.conf
[containers]
log_driver = "journald"
pids_limit = -1
dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
dns_options = ["timeout:2", "attempts:3", "single-request"]

[network]
default_subnet = "172.20.0.0/24"
config.toml — [runners.docker] section
[runners.docker]
  host = "unix:///run/podman/podman.sock"
  privileged = true
  network_mode = "podman"
  network_mtu = 1450
  dns = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
  volumes = ["/etc/containers/executor-containers.conf:/etc/containers/containers.conf:ro", "/cache"]

Notes

  • The IT DNS resolvers (10.130.98.x) above are for us-south. Other regions use
    different addresses — the install script should parameterise these per region:

    Region DNS resolvers
    us-east 10.130.64.4, 10.130.64.70, 10.130.64.134
    us-south 10.130.98.5, 10.130.98.69, 10.130.98.132
    eu-de 10.130.128.5, 10.130.128.69, 10.130.128.133
    lon 10.130.129.4, 10.130.129.68, 10.130.129.132
  • The 172.20.0.0/24 subnet for executor-containers.conf must not conflict with the
    host's podman network (192.168.100.0/24), the PowerVS subnet, or the 10.0.0.0/8
    internal route.

  • All fixes were validated on live dal14 ppc64le runners (aipcc-pro-gl-ppc64le-1..4).
    Post-fix jobs ran successfully including nested podman run workloads
    (bootstrap-and-onboard jobs across all 4 runners, 2 confirmed successe

主要语言
Go
星标
25
派生
41
平均合并
5 天 3 小时
30 天内合并 PR
16

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

redhat-developer/mapt 的其他 Issue

查看 redhat-developer/mapt 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。