Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

`helios / deploy`: switch zone wait in `deploy.sh` has almost no headroom over healthy startup times

Closed Beginner friendly
  • Is something wrong?
#11,453 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust, shell
Domain
ci-cd, devops

Research direction

Start with .github/buildomat/jobs/deploy.sh lines 378–388, where the switch-zone polling limit is set; compare it with the longer Nexus wait mentioned in the issue. The requested change is to give switch-zone startup more headroom. Done when the wait limit is increased and the deploy job no longer times out for the reported startup delays.

Written by the indexing model from the issue text.

Description

Test Flake

Sean's note: I have seen flaky helios jobs fail due to timeout because of switch zone bringup. Below is the analysis done by claude poking at the past few runs on main. I can't spot anything pathological in here, and my conclusion is: we should
probably make this timeout a little bigger.

After installing, .github/buildomat/jobs/deploy.sh polls dendrite in the switch zone (http://[fd00:1122:3344:101::2]:12224/) and gives up after 31 attempts:

https://github.com/oxidecomputer/omicron/blob/main/.github/buildomat/jobs/deploy.sh#L378-L388

With the 1s sleep plus curl overhead, that's about 40 seconds. On main, healthy runs take 24–36 seconds, and the last commit on main (f60eaf3ad5) failed with Failed to reach switch zone after 30 attempts at 40.8s. #11436 hit the same failure at 40.6s, with an almost identical timeline. In both cases dpd started listening about a second after the job gave up.

Where the time goes

Timeline of the failing f60eaf3ad5 run, in seconds after the job starts polling (which is within a second of sled-agent starting). Sources: sled-agent, switch_zone_setup and dendrite SMF logs from the job's artefacts.

Step Time
sled-agent installs the switch zone +4s → +8s (4.1s)
Zone boots to svc:/milestone/single-user +8s → +23s (15.6s)
switch_zone_setup start method +23s → +33s (10s), including two useradds at exactly 4.0s each
sled-agent finishes configuring dendrite (refreshed dendrite service) +33s
dpd start method runs, listening on :12224 +43s
Job gives up +40.8s
Durations on main

The last 18 helios / deploy runs on main, sorted by how long the job waited. Times are relative to the start of the polling loop.

Commit Result Zone boot to single-user Zone initialized Dendrite configured Job waited
ba08500126 passed 6.4s +14s +24s 24.3s
c1d63f4f7e passed 6.3s +15s +25s 24.4s
59f99fea38 passed 6.9s +15s +25s 25.3s
339a8e77ff passed 6.7s +14s +24s 25.3s
654472b636 passed 6.7s +14s +24s 25.3s
cc86abb227 passed 5.9s +13s +24s 25.3s
70eb35226b passed 6.9s +14s +24s 25.3s
52ac94a3c0 passed 6.3s +14s +25s 25.3s
27faee9cf6 passed 6.4s +14s +24s 25.3s
05034a3597 passed 6.0s +14s +24s 25.4s
ea928f8341 passed 5.8s +14s +25s 25.6s
0f2c4a1844 passed 5.6s +13s +24s 33.7s
333d8e4529 passed 5.6s +14s +24s 34.1s
b0d105647a passed 15.9s +23s +34s 35.4s
afad223be0 passed 15.7s +23s +35s 35.4s
d43c79e4d3 passed 16.3s +24s +34s 35.4s
e1c22a131d passed 16.0s +24s +35s 36.4s
f60eaf3ad5 timed out 15.6s +23s +33s 40.8s

Observations:

  • Zone boot is bimodal: 5.6–6.9s or 15.6–16.3s, with nothing in between. A slow boot alone takes a run from ~25s to ~35s.
  • The time after boot is consistent: sled-agent finishes configuring dendrite 10–11s after the zone reaches single-user in every run.
  • dpd sometimes starts late: in most runs, dpd was reachable within about 1s of sled-agent configuring it. In 0f2c4a1844, 333d8e4529 and both failures, it took another 9–10s. In the failing runs, dpd's start method ran at the same moment as zone-network-setup's. zone-network-setup depends on milestone/network, network/routing/ndp and system/manifest-import. The logs don't show which of these was slow.
  • The failures are the runs that hit both delays: a slow boot (+10s) plus a late dpd start (+9s) lands just past the ~40s limit. Either delay alone still passes.
  • useradd takes exactly 4.0s, twice: zone-setup switch-zone (zone-setup/src/switch_zone_user.rs) spends 4.0s in each "Add the user" step, and 2.0s setting up a home directory. Round numbers like this suggest a fixed wait rather than load. The passing runs' switch_zone_setup logs were rotated before upload, so I couldn't compare.
Suggestions
  1. Raise the limit in deploy.sh. The script's other waits (for example, the one for Nexus) allow 300 attempts. A few minutes here would remove the flake without hiding a switch zone that never comes up.
  2. Speed up switch zone startup (separately):
    • Figure out why zone boot sometimes takes ~16s instead of ~6s.
    • Find where the 4.0s per useradd comes from.
    • Find what delays zone-network-setup and dpd after sled-agent configures dendrite.
Dominant language
Rust
Stars
603
Forks
102
Avg merge
3d 23h
Merged PRs (30d)
111

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oxidecomputer/omicron

All issues in oxidecomputer/omicron

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.