foreman-proxy container exhausts its pids limit under remote execution

Open Beginner friendly
  • Is something wrong?
#982 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, ruby
Domain
devops, infrastructure

Research direction

Start by tracing how foremanctl builds and starts the foreman-proxy container, especially where PodmanArgs is configured; the issue reports that --init reaps orphaned processes, while Init=true makes quadlet delete the unit. Check the generated unit and run a proxy through repeated remote executions. Done means the supported configuration starts a real init as PID 1 and the container's pids count returns to baseline after jobs.

Written by the indexing model from the issue text.

Description

The foreman-proxy container never reaps the processes its remote-execution providers create. Each host execution leaves zombies behind for the life of the container, and podman's default pids limit of 2048 is low enough that a busy proxy reaches it after a few hundred host executions. At that point nothing in the container can fork and the proxy stops serving its API.

PID 1 in the container is the Ruby smart-proxy process (Name: smart-proxy, PPid: 0). Both providers start ssh clients that outlive the process that forked them: the script provider runs each execution under unshare --fork --kill-child with ControlMaster=auto and ControlPersist=yes, and the Ansible provider sets the same ControlPersist=yes in ANSIBLE_SSH_ARGS. In both cases the control master detaches, is reparented to PID 1, and is never waited on, so it stays in Z state. pids.current is effectively the zombie count and only grows. Nothing in foremanctl sets a pids limit, so podman's default applies.

The cost is about four pids per host for the script provider and about two for the Ansible provider. Both are cumulative, so the Ansible path reaches the same limit with roughly twice as many hosts — it is half the rate, not immune.

When the limit is reached the container logs:

sh: fork: retry: Resource temporarily unavailable

The proxy stops answering on its API port, and hosts whose content source is that proxy fail undispatched instead of moving to another capsule, so the impact is wider than the hosts the proxy was executing.

On the same deployment, capsules started with --init ran several hundred hosts each at 100%, while capsules with the same pids limit and no init failed every host they were given. The limit was not the variable. The same held for repeated Ansible jobs: the --init proxies accumulated no zombies at all while the default ones grew steadily. Of all containers on the host, only foreman-proxy accumulated zombies in quantity; every other container had none, apart from one service with a handful.

The container needs a real init as PID 1 so that orphaned processes are reaped. Starting it with --init does that: with the equivalent PodmanArgs=--init drop-in, pids return to baseline after each job instead of climbing, and proxies at the default limit execute their full host set. Init=true must not be used — quadlet rejects that key, deletes the unit, and the service does not come back.

Dominant language
Python
Stars
26
Forks
56
Avg merge
2d 18h
Merged PRs (30d)
34

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from theforeman/foremanctl

All issues in theforeman/foremanctl

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.