Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

diff: two large files abort with a ~100 TB allocation

Offen
#287 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
45/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
rust
Bereich
cli

Rechercherichtung

The issue is in src/ed_diff.rs and related files (unified_diff.rs, normal_diff.rs, context_diff.rs, side_diff.rs) where diff::slice is called. First, understand the diff crate's algorithm and its O(n*m) allocation. Look for existing linear-space Myers implementations in Rust or consider patching the diff crate dependency. The goal is to replace the current call with an algorithm that handles large files without excessive memory, and ensure proper error handling instead of aborting. Test with the provided large file example to verify the fix.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

diff builds its line-level LCS with diff::slice() from the diff crate, which allocates an O(n·m) table over the two line vectors. For two
multi-million-line files that table is hundreds of terabytes, the allocation fails, and the process aborts with no diff:-prefixed diagnostic. GNU diff uses a linear-space Myers algorithm and handles the same files in the normal way.

$ python3 -c "open('a','w').write('\n'*5000000)"       # 5 MB, 5M empty lines
$ python3 -c "open('b','w').write('z\n'*5000000)"      # 10 MB, 5M 'z' lines
$ diff a b > /dev/null
memory allocation of 100000040000004 bytes failed
Aborted (core dumped)
$ echo $?
134

Every output mode fails the same way — the allocation happens before any
formatting:

$ for m in "" -u -c -e -y; do diff $m a b >/dev/null; echo "$m -> $?"; done
 -> 134
-u -> 134
-c -> 134
-e -> 134
-y -> 134

Root cause

Each output mode drives the same routine:

// src/ed_diff.rs:74   (identically: unified_diff.rs:68, normal_diff.rs:57,
//                      context_diff.rs:80, side_diff.rs:351)
for result in diff::slice(&expected_lines, &actual_lines) {

diff::slice is the diff crate's LCS over two slices; its dynamic-programming table is proportional to expected_lines.len() * actual_lines.len(). With 5M lines on each side that is ~2.5·10¹³ cells — the observed request is 100,000,040,000,004 bytes (~100 TB). Nothing bounds the input size before the call, and the allocation failure is an abort rather than an error the caller can report.

Vorherrschende Sprache
Rust
Sterne
276
Forks
39
Ø Merge
4 T. 12 Std.
Gemergte PRs (30 T.)
3

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus uutils/diffutils

Alle Issues in uutils/diffutils

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.