Skip to content

fix(ibmcloud): remove PID limits to prevent EAGAIN during heavy builds#856

Open
adrianriobo wants to merge 1 commit into
redhat-developer:mainfrom
adrianriobo:fix-ibm-limits
Open

fix(ibmcloud): remove PID limits to prevent EAGAIN during heavy builds#856
adrianriobo wants to merge 1 commit into
redhat-developer:mainfrom
adrianriobo:fix-ibm-limits

Conversation

@adrianriobo

@adrianriobo adrianriobo commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

On ppc64le (and s390x) runners, building wheel packages fails with:

```
ccache: error: Failed to fork: Resource temporarily unavailable
gcc: error: posix_spawn: Operation not permitted
```

Root cause: Pillow compiles 60+ C extension files. Each file spawns a
`ccache → gcc → cc1 → as` process chain (4 processes per file). Running
two concurrent jobs exhausts the container's cgroup `pids.max`, which
Podman inherits from the system default (~2048 on RHEL 9).

Three enforcement layers were all unset, each independently capable of
triggering the failure:

Layer Before After
Podman `pids_limit` in `containers.conf` system default `-1` (unlimited)
Docker executor `--docker-pids-limit` not set (inherits Podman default) `-1` (unlimited)
Systemd `TasksMax` on `gitlab-runner.service` systemd default (512–4096) `infinity`

Changes

  • `pkg/integrations/gitlab/snippet-linux.sh`: adds a `pids_limit = -1`
    block in `/etc/containers/containers.conf` (idempotent, covers all DNS
    code-path branches); adds `--docker-pids-limit -1` to `gitlab-runner register`
    so the Docker executor explicitly removes the pids cgroup on each job container.
  • `pkg/provider/ibmcloud/action/ibm-power/cloud-config`: adds
    `TasksMax=infinity` to the `gitlab-runner.service.d` drop-in.
  • `pkg/provider/ibmcloud/action/ibm-z/cloud-config`: same fix applied to
    IBM Z runners to prevent the same failure class on s390x.

Test plan

  • Re-provision a ppc64le runner with these changes and trigger a Pillow wheel build
  • Confirm no EAGAIN / posix_spawn errors during C extension compilation
  • Verify s390x runner is unaffected (TasksMax=infinity is a no-op on non-constrained hosts)

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Configuration changes remove container PID limits and systemd task limits for GitLab runners. The GitLab integration script updates Podman and Docker runner settings, while IBM Power and IBM Z templates set TasksMax=infinity for the runner service.

Changes

PID and Task Limit Removal

Layer / File(s) Summary
Container PID limit configuration
pkg/integrations/gitlab/snippet-linux.sh
Ensures Podman uses pids_limit = -1 in the [containers] section and inserts the same setting into the Docker executor configuration.
Systemd TasksMax override
pkg/provider/ibmcloud/action/ibm-power/cloud-config, pkg/provider/ibmcloud/action/ibm-z/cloud-config
Adds TasksMax=infinity to the GitLab runner systemd service override on IBM Power and IBM Z.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly describes removing PID limits to fix EAGAIN issues during heavy builds.
Description check ✅ Passed The description matches the PR by explaining the PID-limit failures and the container, runner, and systemd changes.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

On ppc64le (and s390x) runners, compiling Pillow's 60+ C extension files
causes ccache->gcc->cc1->as process chains that exhaust the container's
cgroup pids.max, producing EAGAIN on fork and posix_spawn failures.

Three enforcement layers are now cleared on every GitLab runner setup:

- snippet-linux.sh: sets pids_limit = -1 in /etc/containers/containers.conf
  so Podman does not apply a default ceiling to job containers.
- snippet-linux.sh: passes --docker-pids-limit -1 to gitlab-runner register
  so the Docker executor explicitly removes the pids cgroup on each job
  container it spawns.
- ibm-power/cloud-config, ibm-z/cloud-config: adds TasksMax=infinity to
  the gitlab-runner.service.d drop-in so the runner daemon's own systemd
  task limit does not cap its child process tree.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Xaenalt

Xaenalt commented Jul 14, 2026

Copy link
Copy Markdown

So far, PID issue has not resurfaced, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants