bug: agent bootstrap script can't overwrite in-use coder binary (curl error 23 / ETXTBSY)

从 provisionersdk/scripts/bootstrap_linux.sh 开始，检查下载循环以及其中的 curl、wget 和 busybox wget 分支。使用正在使用中的 coder 二进制文件复现持久化 BINARY_DIR 的情况，然后验证 bootstrap 能从下载失败中恢复，而不是无限重试。

Summary

When the agent bootstrap script runs against a BINARY_DIR that persists across workspace restarts (notably envbox, which sets BINARY_DIR=$HOME/.coder — see envbox/cli/docker.go), curl can fail to overwrite the existing coder binary if anything still holds a reference to it. This produces curl: (23) Failure writing output to destination and the bootstrap loop never recovers.

Reproduction

1. Run a workspace using envbox (sysbox-based). Inner-container workload doesn't strictly matter, but heavy/leaky workloads such as a kind cluster running Argo/Grafana/Loki/Mimir/Tempo make it much more likely to trigger.
2. Force-delete the workspace pod without graceful shutdown (e.g. kubectl delete pod ... --force --grace-period=0), simulating a stuck/unresponsive workspace.
3. Restart the workspace.
4. The bootstrap script enters the curl retry loop and never recovers:

+ curl -fsSL --compressed https://<access-url>/bin/coder-linux-amd64 -o coder
curl: (23) Failure writing output to destination
+ status=23
+ echo error: failed to download coder agent
+ echo    command returned: 23
+ echo Trying again in 30 seconds...
+ sleep 30

Root cause

provisionersdk/scripts/bootstrap_linux.sh does:

curl -fsSL --compressed "${BINARY_URL}" -o "${BINARY_NAME}" && break

without first removing the existing destination file. When BINARY_DIR persists and the old coder binary still has an open reference (e.g. ETXTBSY — "text file busy" — from a process the orphaned/force-killed pod left behind), the kernel refuses to overwrite the running executable and curl returns 23 indefinitely.

The workaround that resolves it is manually running rm -f $HOME/.coder/coder inside the inner container before retrying.

Suggested fix

In bootstrap_linux.sh, before the download loop, either:

• rm -f "${BINARY_NAME}", or
• download to a temp filename and mv into place (atomic replace, also avoids partial-download issues).

The same treatment should be applied to the wget / busybox wget branches.

Impact

Workspaces relying on a persistent BINARY_DIR (envbox is the main case) can become permanently stuck in the bootstrap retry loop after any unclean shutdown, with no actionable error surfaced in the UI. The only recovery today is exec-ing into the inner container and removing the binary by hand.

Created on behalf of @ericpaulsen.