A drain can deadlock: a running CI job's new lease is held until the restart, and the restart waits for the job

Open
  • Is something wrong?
#116 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
swift
Domain
devops, distributed-systems

Research direction

Start by tracing admit, RestartPlan.granted, and the lease handling for CI job holders in the drain flow; the payload does not name source files or test paths. Reproduce the pending restart with a held runner and a job requesting another lease, then check whether the request is granted or refused promptly and the restart completes when the job ends. Also inspect oss restart status so it reports what the drain waits on and for how long.

Written by the indexing model from the issue text.

Description

Seen 2026-11-04: an install's drain waited 30+ minutes on two of Isomer's CI leases, a runner container and a mac slot. Their jobs were in platform tests that lease more machines (Android, Wine, Apple) as they go. While a restart is pending, `admit` holds every new lease but a session's (`RestartPlan.granted`). So each job waited for a lease the drain wouldn't give, and the drain waited for the jobs. It took the owner cancelling the runs to break it.

Wanted, either of:

  • A running CI job's own leases are granted during a drain (holders `-@ci` whose runner is held), as a session's are, so the job can finish. Bounded by a drain deadline after which nothing new is granted.
  • Or a request that arrives during a drain fails fast (`restarting`, retry after), not waiting in line, so the job fails and ends, and the drain completes. The job is rerun after.

The first finishes more work; the second is simpler. Also wanted: `oss restart status` names what the drain waits on and how long, so an installer can see a deadlock instead of waiting.

Tests: with a restart pending, a held runner's job's lease is granted (or refused at once), and the restart completes once the job ends.

Dominant language
Swift
Stars
1
Forks
0
Avg merge
2h 28m
Merged PRs (30d)
113

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from AttilaTheFun/oss

All issues in AttilaTheFun/oss

Similar issues

More Swift issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.