Concurrent cache hits for the same task fail to restore outputs with "File exists (os error 17)"
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 48/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- node.js, rust
- 领域
- build-system
调研方向
Start with crates/vt/src/session/cache/archive.rs, especially extract_output_archive, and compare it with the restore flow in crates/vt/src/session/cache/remote.rs. The issue describes both concurrent restores and readers observing files during restore; inspect how local and remote cache hits reach extraction and how files are replaced. Done means concurrent hits complete successfully and readers do not observe missing or partial outputs; the issue asks to agree on lock scope before a PR.
由索引模型根据 Issue 内容生成。
描述
When two vp run processes get a cache hit for the same task at the same time, both extract the output archive over the workspace. tar-rs opens each file with create_new, and when the file exists it removes it and tries once more (entry.rs#L646-L661, same code in 0.4.46, which vite-plus 1.1.0 ships). The two processes race on every path, and one of them usually stops partway. With vite-plus 1.1.0:
$ node gen.mjs ◉ cache hit, replaying
✗ Cache lookup failed: failed to restore cached outputs from …/node_modules/.vite/task-cache/v18/….tar.zst; the archive may have been deleted or corrupted. Run `vp cache clean` to clear the cache.: failed to unpack `…/out/1017.txt`: failed to unpack `out/1017.txt` into `…/out/1017.txt`: File exists (os error 17)
---
vp run: cache hit, 185ms saved.
The archive is fine and the other process completes the restore, so the files end up correct. In vite-plus 1.1.0 the failed run still exits 0 and counts as a hit (#767). On main, after #770, the same race fails the run with exit 1 and suggests cleaning the cache, though nothing in it is wrong. With vt built from 9ba4cc9:
$ node gen.mjs ◉ cache hit, replaying
✗ Cache restore failed. Run `vt cache clean` to clear the cache: failed to extract the output archive: failed to unpack `…/out/10.txt`: failed to unpack `out/10.txt` into `…/out/10.txt`: File exists (os error 17)
Readers are affected even without a second restore. Every hit removes and recreates each output, including files whose content didn't change, so a process reading those outputs while one vp run restores them can find a file missing or shorter than it should be. #822 mentions this as a cost of running one vp run per package.
Our pre-push hook used to run five vp run scripts in parallel, and three of them start with the same codegen task (vp run i18n:generate && ...). We now finish two of those before starting the rest.
Steps to reproduce
package.json:
{ "name": "vt-restore-race", "private": true, "type": "module", "devDependencies": { "vite-plus": "1.1.0" } }
vite.config.ts:
import { defineConfig } from 'vite-plus'
export default defineConfig({
run: {
tasks: {
gen: {
command: 'node gen.mjs',
cache: { input: ['gen.mjs'], output: ['out/**'] },
},
},
},
})
gen.mjs:
import { mkdirSync, writeFileSync } from 'node:fs'
mkdirSync('out', { recursive: true })
for (let i = 0; i < 2000; i++) writeFileSync(`out/${i}.txt`, 'x'.repeat(4096))
pnpm install
node_modules/.bin/vp run gen # populate the cache
for i in $(seq 1 30); do
(node_modules/.bin/vp run gen > a$i.log 2>&1; echo "exit $?" >> a$i.log) &
(node_modules/.bin/vp run gen > b$i.log 2>&1; echo "exit $?" >> b$i.log) &
wait
done
grep -l 'os error 17' *.log | wc -l # failed restores
grep -L 'exit 0' *.log | wc -l # runs with a non-zero exit
With vite-plus 1.1.0 this printed 30 and 0: in every pair one run failed, never both, and every run exited 0. The same loop with vt built from 9ba4cc9 and the task in vite-task.json printed 30 and 30. Node 24.21.0, macOS 27.0.1 arm64. I haven't tried Linux or Windows. The restore path in vite-task has no platform-specific code, though the error may differ on Windows.
Reader check
reader.mjs:
import { readFileSync } from 'node:fs'
const end = Date.now() + 1500
let missing = 0
let short = 0
while (Date.now() < end) {
for (let i = 0; i < 2000; i += 7) {
try {
if (readFileSync(`out/${i}.txt`).length !== 4096) short++
} catch (e) {
if (e.code === 'ENOENT') missing++
else throw e
}
}
}
console.log({ missing, short })
node reader.mjs & sleep 0.2; node_modules/.bin/vp run gen; wait
With only one vp run restoring, 5 runs on vite-plus 1.1.0 printed missing between 1 and 6 and short 0. With a debug vt build from 9ba4cc9, which restores more slowly, 3 runs printed missing 2 to 4 and short 1 to 4.
Expected: both runs restore the outputs and exit 0, and outputs don't disappear or show partial content while another process reads them.
The restore path takes no lock. The only explicit lock between processes is db_open.lock, held while the database opens (mod.rs#L295-L313), and extract_output_archive unpacks straight into the workspace root (archive.rs#L56-L66). A lock around extraction of the same local archive would fix this reproducer, and the process that gets it second could skip files that already match. Remote hits need a shared key too: each download gets a new archive file name (remote.rs#L204), even when two processes restore the same task. The reader case needs each file swapped in one step, for example by unpacking to a temporary file next to it and renaming it into place. I'd like to agree on the lock scope before working on a PR.
- 主要语言
- Rust
- 星标
- 473
- 派生
- 44
- 平均合并
- 1 天 4 小时
- 30 天内合并 PR
- 73
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
voidzero-dev/vite-task 的其他 Issue
-
Windows: automatic filesystem tracking makes Git/MSYS sh.exe fail with MapViewOfFileEx error 487未关闭
难度 5/5 一周以上 新手友好度 18/100
voidzero-dev/vite-task#830 ·
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 35/100
voidzero-dev/vite-task#822 · 1 条评论 · 1 个 reaction ·
维护者通常 1 天内回复
-
Let tools mark cache directories with CACHEDIR.TAG可能已有人在做 @lifeiscontent 于 10 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 35/100
voidzero-dev/vite-task#793 ·
维护者通常 1 天内回复
-
Preview cache hits without running tasks可能已有人在做 @lifeiscontent 于 11 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 52/100
voidzero-dev/vite-task#791 ·
维护者通常 1 天内回复
-
Built-in tool tasks miss the cache after the workspace moves可能已有人在做 @lifeiscontent 于 11 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 67/100
voidzero-dev/vite-task#790 ·
维护者通常 1 天内回复
查看 voidzero-dev/vite-task 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 65/100
storytold/photocraft#5260 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
-
难度 2/5 1-3 小时 新手友好度 68/100
-
难度 2/5 1-3 小时 新手友好度 62/100
extendr/extendr.github.io#134 ·
-
awaiting-response bug
难度 1/5 1 小时以内 新手友好度 72/100
维护者通常 1 天内回复