bug: agent bootstrap script can't overwrite in-use coder binary (curl error 23 / ETXTBSY)

provisionersdk/scripts/bootstrap_linux.sh から始め、ダウンロードループと、その curl、wget、busybox wget の各分岐を調査します。使用中の coder バイナリがある永続的な BINARY_DIR のケースを再現し、その後、bootstrap がダウンロード失敗から回復し、無限に再試行し続けないことを確認します。

Summary

When the agent bootstrap script runs against a BINARY_DIR that persists across workspace restarts (notably envbox, which sets BINARY_DIR=$HOME/.coder — see envbox/cli/docker.go), curl can fail to overwrite the existing coder binary if anything still holds a reference to it. This produces curl: (23) Failure writing output to destination and the bootstrap loop never recovers.

Reproduction

1. Run a workspace using envbox (sysbox-based). Inner-container workload doesn't strictly matter, but heavy/leaky workloads such as a kind cluster running Argo/Grafana/Loki/Mimir/Tempo make it much more likely to trigger.
2. Force-delete the workspace pod without graceful shutdown (e.g. kubectl delete pod ... --force --grace-period=0), simulating a stuck/unresponsive workspace.
3. Restart the workspace.
4. The bootstrap script enters the curl retry loop and never recovers:

+ curl -fsSL --compressed https://<access-url>/bin/coder-linux-amd64 -o coder
curl: (23) Failure writing output to destination
+ status=23
+ echo error: failed to download coder agent
+ echo    command returned: 23
+ echo Trying again in 30 seconds...
+ sleep 30

Root cause

provisionersdk/scripts/bootstrap_linux.sh does:

curl -fsSL --compressed "${BINARY_URL}" -o "${BINARY_NAME}" && break

without first removing the existing destination file. When BINARY_DIR persists and the old coder binary still has an open reference (e.g. ETXTBSY — "text file busy" — from a process the orphaned/force-killed pod left behind), the kernel refuses to overwrite the running executable and curl returns 23 indefinitely.

The workaround that resolves it is manually running rm -f $HOME/.coder/coder inside the inner container before retrying.

Suggested fix

In bootstrap_linux.sh, before the download loop, either:

• rm -f "${BINARY_NAME}", or
• download to a temp filename and mv into place (atomic replace, also avoids partial-download issues).

The same treatment should be applied to the wget / busybox wget branches.

Impact

Workspaces relying on a persistent BINARY_DIR (envbox is the main case) can become permanently stuck in the bootstrap retry loop after any unclean shutdown, with no actionable error surfaced in the UI. The only recovery today is exec-ing into the inner container and removing the binary by hand.

Created on behalf of @ericpaulsen.