fs.DiskUsage fails when a file is deleted during the walk (lstat/readdirent: no such file or directory)
还没有人认领这个 Issue。
评估
调研方向
在引用的 callback 位置阅读 fs/du_unix.go 和 fs/du_windows.go，然后检查它们的 walk 测试如何覆盖遍历过程中消失的路径。复现 issue 中描述的 ENOENT 情况，并检查这两个文件能否处理这些情况，同时确保 root 缺失时仍会返回错误。当条目在遍历过程中消失时，磁盘使用量计算仍能完成，并且仍会报告 root 缺失，即表示工作完成。
由索引模型根据 Issue 内容生成。
描述
Problem
fs.DiskUsage fails if a file under the root is deleted while it walks. On Unix it is a filepath.Walk whose callback returns the error it receives (fs/du_unix.go#L68-L71):
if err := filepath.Walk(root, func(path string, fi os.FileInfo, err error) error {
if err != nil {
return err
}
filepath.Walk reads a directory and then lstats each entry. If the entry is removed between those two calls, the callback gets lstat …: no such file or directory and the whole usage call fails. A directory removed between being listed and being read fails with readdirent …: no such file or directory.
That is the normal case for the main caller. The containerd overlay snapshotter computes Usage of an active snapshot (a running container's writable layer) with fs.DiskUsage(ctx, upperPath), and running containers create and delete temp files all the time. In Docker (containerd image store), docker ps --size, docker ps --format '{{json .}}' and docker system df then fail:
Error response from daemon: snapshotter.Usage failed for db1ad9bf…: lstat /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/25/fs/tmp/cc/f17: no such file or directory
Error response from daemon: snapshotter.Usage failed for db1ad9bf…: readdirent /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/25/fs/tmp/cc: no such file or directory
Reproduce
With Docker on the containerd image store (continuity v0.5.0 via containerd v2.4.1, Docker 29.9.0):
docker run -d --name churn alpine:3.20 sh -c 'while true; do mkdir -p /tmp/cc; for i in $(seq 300); do : > /tmp/cc/f$i; done; rm -rf /tmp/cc; done'
sleep 2
for i in $(seq 50); do docker ps -a --size >/dev/null 2>&1 || echo fail; done | wc -l # 16 of 50 failed
docker rm -f churn
Without Docker: this Go program copies the diskUsage walk, runs it while a goroutine creates and removes 300 files in a subdirectory, and compares it with the fix below (Go 1.27.1, Linux 6.8, ext4):
continuity v0.5.0 walk: failed 1060/2000 map[lstat <path>: no such file or directory:1050 readdirent <path>: no such file or directory:10]
skip-ENOENT walk: failed 0/2000 map[]
missing root still fails: true
Suggested fix
A disk-usage walk is a point-in-time estimate, so an entry that disappears mid-walk has no size to count. Skip it instead of failing:
if err := filepath.Walk(root, func(path string, fi os.FileInfo, err error) error {
if err != nil {
if os.IsNotExist(err) && path != root {
return nil // removed while walking; it no longer uses space
}
return err
}
This also covers the directory case: when reading a directory fails, filepath.Walk passes that error to the callback for the directory, and returning nil skips the directory (the readdirent error is an *os.PathError wrapping ENOENT, so os.IsNotExist matches it). Keep failing if root itself is missing, so a wrong path is still reported.
fs/du_windows.go has the same callback and needs the same change.
Workarounds
Callers can avoid the walk when they don't need sizes: docker ps with explicit --format fields or --size=false (docker/cli#4067), or GET /system/df?type=image for image usage. A caller that needs the size can retry on ENOENT.
Related
- Docker: the daemon fails the whole container list on this error (the graphdriver store logs it and reports
-1). - The same error in other projects: ptone/scion#1867, ptone/scion#1864, openzim/zimfarm#1591, link-assistant/hive-mind#3000.
- containerd/nerdctl#3027: snapshotter
Usagecalls are expensive with many objects.
- 主要语言
- Go
- 星标
- 153
- 派生
- 74
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
containerd/continuity 的其他 Issue
-
难度 5/5 一周以上 新手友好度 25/100
containerd/continuity#277 · 1 条评论 ·
-
nerdctl commit taking long time possibly because of fs.Changes walking through all files可能已有人在做 @gogasca 于 257 天前认领。 未关闭enhancement
难度 4/5 3-5 天 新手友好度 38/100
containerd/continuity#236 · 5 条评论 ·
-
`continuity.context.Walk` assumes incorrect behavior for `os.Readlink`可能已有人在做 @sueun-dev 于 58 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 45/100
containerd/continuity#232 ·
-
难度 3/5 1-2 天 新手友好度 35/100
containerd/continuity#226 · 1 条评论 ·
-
难度 4/5 3-5 天 新手友好度 35/100
containerd/continuity#221 ·
查看 containerd/continuity 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 86/100
oalders/clodhopper#162 ·
维护者通常 1 天内回复
-
component/server severity/minor type/bug
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复