[Bug] health_cyborg_check reports healthy while cyborg-agent is down

Closed Beginner friendly
  • Is something wrong?
#1,708 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@arasHi87 is already working on this.

Since Oct 9, 2026.

  • #1819 by @SekiXu — merged

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
shell
Domain
observability

Research direction

Start with health_cyborg_check in core/sdk_sh/modules/sdk_health.sh and compare the compute-node branch with the cyborg-api and cyborg-conductor branches. Check how ERR_CODE and ERR_LOG feed into auto-repair. Done means the check reports code 5 and the node's journal when cyborg-agent is stopped, auto-repair restarts it, and the check reports healthy when all Cyborg services are running.

Written by the indexing model from the issue text.

Description

Description

health_cyborg_check in core/sdk_sh/modules/sdk_health.sh misses a stopped cyborg-agent on a compute node. The compute loop sets a lowercase err_code=5, but the health framework reads ERR_CODE. So the check:

  • reports Cyborg healthy (cluster check passes) while an agent is down
  • never triggers _health_cyborg_auto_repair, which only runs when ERR_CODE is non-zero, so the agent is never restarted
  • sets no ERR_LOG, unlike the cyborg-api and cyborg-conductor branches
    for node in "${CUBE_NODE_COMPUTE_HOSTNAMES[@]}" ; do
        if ! is_remote_running $node cyborg-agent ; then
            ERR_MSG+="cyborg-agent on $node is not running\n"
            err_code=5 && break        # <- should be ERR_CODE=5 (and set ERR_LOG)
        fi
    done

Found by reading the code (origin/develop @ ebda9261) while planning the 3.2.0 upgrade checks. Not reproduced on a cluster yet.

How to Reproduce

  1. On a compute node: systemctl stop cyborg-agent
  2. On a control node: hex_cli -c cluster check Compute (or hex_sdk health_cyborg_report)
  3. Cyborg reports healthy

Expected behavior

The check fails with error code 5 and the agent's journal as ERR_LOG. Auto-repair restarts cyborg-agent.

Additional Context (Optional)

GPU VMs that use pGPU passthrough depend on cyborg-agent, so a silently dead agent breaks GPU VM creation with no health signal. Related known-issue: cyborg-boot-race-burns-systemd-start-limit.

QA Verification

# Command / steps Expected outcome
1 Stop cyborg-agent on a compute node, then run hex_sdk health_cyborg_report reports an error (code 5) naming the node
2 Leave it stopped for the auto-repair rounds cyborg-agent is restarted and a "fixing completed" event is logged
3 With every Cyborg service running, run hex_sdk health_cyborg_report healthy
  • QA Status declared at Done — when this issue reaches Done, set the board's
    QA Status field to Not needed or Ready to QA. Never leave it empty on a Done
    ticket; In QA / Verified are QA's own transitions.

Output artifacts (Definition of Done)

Beyond code, docs, and config changes, completing this issue must also deliver:

  • Handbook knowledge update — land the durable, team-readable knowledge from this
    work into the bigstack-handbook cubecos kb (kb/cubecos/…) via
    /bigstack-core:save-to-handbook (Topic / Runbook / Known-issue / ADR as fits).
    Diagnosed or fixed a failure? Ship the scenario sidecar (<note>.scenario.yaml)
    beside the note.
Dominant language
Shell
Stars
161
Forks
19
Avg merge
1d 13h
Merged PRs (30d)
132

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from bigstack-oss/cubecos

All issues in bigstack-oss/cubecos

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.