Failing steps lose $GITHUB_OUTPUT and $GITHUB_ENV in novolume mode

Open
#461 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
66/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, typescript
Domain
backend, devops

Research direction

Start with packages/k8s/src/hooks/run-script-step.ts, especially the runScriptStep error path around execPodStep and execCpFromPod; compare the behavior with the novolume package change mentioned in issue #445. Reproduce the failed-step case with the provided workflow and check whether output, environment, path, and summary files are copied back after failure. Done means copy failures are logged without hiding the original step error, with a deadline on that failure-path copy.

Written by the indexing model from the issue text.

Description

If a run: step writes to $GITHUB_OUTPUT and then exits non-zero, the runner never sees what it wrote. Same for $GITHUB_ENV, $GITHUB_PATH and $GITHUB_STEP_SUMMARY. On GitHub-hosted runners the output is there. Nothing is logged about the drop either, so it's easy to miss.

Setup: ARC gha-runner-scale-set chart 0.14.2, runner 2.338.2, /home/runner/k8s-novolume from a build of main after v0.8.1 (same code for this function as v0.8.1), job has a container:.

on: workflow_dispatch
jobs:
  repro:
    runs-on: <novolume scale set>
    container:
      image: <any image with bash>
    steps:
      - id: failing
        continue-on-error: true
        run: |
          echo "a=written-before-failure" >> "$GITHUB_OUTPUT"
          exit 1
      - id: control
        run: echo "c=written-then-exit-0" >> "$GITHUB_OUTPUT"
      - run: |
          echo "failing.outputs.a='${{ steps.failing.outputs.a }}'"
          echo "control.outputs.c='${{ steps.control.outputs.c }}'"

I get failing.outputs.a='' and control.outputs.c='written-then-exit-0'. The failing step's log ends with Error: failed to run script step: command terminated with non-zero exit code: command terminated with exit code 1. The control step in the same pod is fine, so the copy-back itself works.

In runScriptStep the catch around execPodStep throws before the execCpFromPod(... _runner_file_commands ...) call, which sits after the try/catch: https://github.com/actions/runner-container-hooks/blob/v0.8.1/packages/k8s/src/hooks/run-script-step.ts#L77-L106. main has the same code for this function, apart from how the error message is formatted.

The fix I'd try is doing the copy on the failure path too, before throwing the step error, and warning instead of throwing if the copy fails so it doesn't hide the real error. execCpFromPod has no overall timeout, so I'd put a deadline on it in that path.

#445 looks like it already copies after a failed step in the novolume package. Is that on purpose? I can send a PR for this against main, but #445 moves the file.

Dominant language
TypeScript
Stars
156
Forks
122
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from actions/runner-container-hooks

All issues in actions/runner-container-hooks

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.