diff: two large files abort with a ~100 TB allocation
还没有人认领这个 Issue。
评估
调研方向
The issue is in src/ed_diff.rs and related files (unified_diff.rs, normal_diff.rs, context_diff.rs, side_diff.rs) where diff::slice is called. First, understand the diff crate's algorithm and its O(n*m) allocation. Look for existing linear-space Myers implementations in Rust or consider patching the diff crate dependency. The goal is to replace the current call with an algorithm that handles large files without excessive memory, and ensure proper error handling instead of aborting. Test with the provided large file example to verify the fix.
由索引模型根据 Issue 内容生成。
描述
diff builds its line-level LCS with diff::slice() from the diff crate, which allocates an O(n·m) table over the two line vectors. For two
multi-million-line files that table is hundreds of terabytes, the allocation fails, and the process aborts with no diff:-prefixed diagnostic. GNU diff uses a linear-space Myers algorithm and handles the same files in the normal way.
$ python3 -c "open('a','w').write('\n'*5000000)" # 5 MB, 5M empty lines
$ python3 -c "open('b','w').write('z\n'*5000000)" # 10 MB, 5M 'z' lines
$ diff a b > /dev/null
memory allocation of 100000040000004 bytes failed
Aborted (core dumped)
$ echo $?
134
Every output mode fails the same way — the allocation happens before any
formatting:
$ for m in "" -u -c -e -y; do diff $m a b >/dev/null; echo "$m -> $?"; done
-> 134
-u -> 134
-c -> 134
-e -> 134
-y -> 134
Root cause
Each output mode drives the same routine:
// src/ed_diff.rs:74 (identically: unified_diff.rs:68, normal_diff.rs:57,
// context_diff.rs:80, side_diff.rs:351)
for result in diff::slice(&expected_lines, &actual_lines) {
diff::slice is the diff crate's LCS over two slices; its dynamic-programming table is proportional to expected_lines.len() * actual_lines.len(). With 5M lines on each side that is ~2.5·10¹³ cells — the observed request is 100,000,040,000,004 bytes (~100 TB). Nothing bounds the input size before the call, and the allocation failure is an abort rather than an error the caller can report.
- 主要语言
- Rust
- 星标
- 276
- 派生
- 39
- 平均合并
- 4 天 12 小时
- 30 天内合并 PR
- 3
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
uutils/diffutils 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 70/100
-
难度 2/5 1-3 小时 新手友好度 85/100
-
难度 2/5 1-3 小时 新手友好度 72/100
-
难度 2/5 1-3 小时 新手友好度 68/100
-
难度 3/5 1-2 天 新手友好度 65/100
相似的 Issue
-
bug
难度 1/5 1 小时以内 新手友好度 85/100
-
难度 2/5 1-3 小时 新手友好度 75/100
yantrikos/yantrik-os#255 ·
-
bug CLI custom-model
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 70/100
raphamorim/rio#1956 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
rust-bitcoin/rust-bitcoin#6930 · 1 条评论 ·