CI: a package-server outage fails the valgrind and bench checks in their setup, and the merge queue ejects every PR in turn

Open
  • Is something wrong?
#2,341 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
c, docker, github-actions, ubuntu
Domain
ci-cd, infrastructure

Research direction

Start by reading tools/install_valgrind.sh and the workflow definitions for valgrind (memcheck smoke, JIT off) and bench (instruction-count regression gate). Trace how the other container jobs get their dependencies, then choose an approach that avoids package-server access when these jobs start. Done means both checks start without a package server and remain green when name resolution is unavailable.

Written by the indexing model from the issue text.

Description

area:ci found-by:ci

On 2026-11-11 the merge queue ejected three green pull requests in fifteen minutes (#2338 at 14:30Z, #2336 at 14:35Z, #2337 at 14:40Z), each with failed_checks. No change was at fault. In the first merge group (8559c7b9) two required checks failed in their setup step:

  • valgrind (memcheck smoke, JIT off)
  • bench (instruction-count regression gate)

Both install valgrind with apt when the job starts (tools/install_valgrind.sh). The runner could not resolve the package servers:

Err:2 http://security.ubuntu.com/ubuntu resolute-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
##[error]SETUP FAILURE — could not install valgrind after 3 bounded attempts. This is NOT a regression detected by the gate; the gate never ran.

The script says rightly that the gate never ran. The queue cannot tell: a required check is red, so the pull request leaves the queue, the next group is built, and it fails the same way.

What would fix it

Do not fetch packages when the job starts. Either the two jobs run in the dev image with valgrind installed in it, as the other container jobs do, or the runner gets valgrind from a cache that the job restores. Three retries inside one outage do not help; the outage lasted longer than the retries.

Done when

  • Neither job needs a package server to start.
  • A planted failure of name resolution in the job (an unreachable mirror) no longer turns the check red.

— Claude DevBox

Dominant language
C
Stars
3
Forks
7
Avg merge
6h 32m
Merged PRs (30d)
285

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from InauguralSystems/EigenScript

All issues in InauguralSystems/EigenScript

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.