[FR] Support zoekt indexing of binary files
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Anfängerfreundlichkeit
- 25/100
Rechercherichtung
Start with IndexBuilder.Add and DocChecker.Check in indexbuilder.go, especially their null-byte checks and skip-marker handling. First determine the intended behavior for PDFs and other binary files; done should include an agreed indexing approach and tests showing the desired result without incorrectly treating binary content as text.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
When trying to index binary files (like PDFs), they are skipped with the following reason: binary content at byte offset x. Seems like binary files with a byte offset don't play nice with zoekt for whatever reason.
Diagnosing with Ask:
You're absolutely correct! The primary reason PDFs and other binary files are skipped is because they contain null bytes (0x00), not just because of size limits or trigram counts.
The Null Byte Check
In IndexBuilder.Add() indexbuilder.go:428-431:
if idx := bytes.IndexByte(doc.Content, 0); idx >= 0 {
doc.SkipReason = fmt.Sprintf("binary content at byte offset %d", idx)
doc.Language = "binary"
}
This checks if there's a null byte anywhere in the file content. If found, it:
- Sets the
SkipReasonto indicate where the binary content was detected - Marks the language as "binary"
Then the content is replaced indexbuilder.go:433-440:
if doc.SkipReason != "" {
doc.Content = []byte(notIndexedMarker + doc.SkipReason)
doc.Symbols = nil
doc.SymbolsMetaData = nil
if doc.Language == "" {
doc.Language = "skipped"
}
}
The file content is replaced with a marker string like "NOT-INDEXED: binary content at byte offset 5" indexbuilder.go:386.
Similar Check in DocChecker
The DocChecker.Check() method has an identical check indexbuilder.go:544-546:
if index := bytes.IndexByte(content, 0); index > 0 {
return fmt.Errorf("binary data at byte offset %d", index)
}
Bottom line: PDFs cannot be indexed in Zoekt because they contain null bytes, which are fundamental to binary file formats. The presence of even a single null byte causes the file to be marked as binary and excluded from text-based indexing. There's no configuration option to override this behavior since Zoekt is designed specifically for text-based source code search.
- Vorherrschende Sprache
- TypeScript
- Sterne
- 3.9k
- Forks
- 374
- Ø Merge
- 21 Std. 18 Min.
- Gemergte PRs (30 T.)
- 39
Beitragsleitfaden
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus sourcebot-dev/sourcebot
-
bug triage needed
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 84/100
sourcebot-dev/sourcebot#1681 · 1 Kommentar ·
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 84/100
sourcebot-dev/sourcebot#1633 ·
-
bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
sourcebot-dev/sourcebot#1384 · 4 Kommentare ·
-
ask_sb bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
sourcebot-dev/sourcebot#1137 · 4 Kommentare ·
-
bug triage needed
Schwierigkeit 3/5 1-2 Tage Anfängerfreundlichkeit 48/100
sourcebot-dev/sourcebot#1661 · 1 Kommentar ·
Alle Issues in sourcebot-dev/sourcebot
Ähnliche Issues
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 75/100
mksglu/context-mode#1200 ·
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 75/100
jaegertracing/jaeger-ui#4506 ·
-
area:desktop area:ui bug platform:macos
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 75/100
anthropics/claude-code#96687 ·
-
good first issue
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 95/100
AOSSIE-Org/DebateAI#582 · 2 Kommentare ·