[FR] Support zoekt indexing of binary files

Ouverte

#575 ouverte le 27/10/2025 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté: 5/5
Temps estimé: Plus d'une semaine
Accessibilité débutants: 25/100

Type d'issue: Fonctionnalité
Clarté: À clarifier
Activité: À l'abandon
Stack technique: go
Domaine: search

Piste de recherche

Start with IndexBuilder.Add and DocChecker.Check in indexbuilder.go, especially their null-byte checks and skip-marker handling. First determine the intended behavior for PDFs and other binary files; done should include an agreed indexing approach and tests showing the desired result without incorrectly treating binary content as text.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Feature zoekt

When trying to index binary files (like PDFs), they are skipped with the following reason: binary content at byte offset x. Seems like binary files with a byte offset don't play nice with zoekt for whatever reason.

Diagnosing with Ask:

You're absolutely correct! The primary reason PDFs and other binary files are skipped is because they contain null bytes (0x00), not just because of size limits or trigram counts.

The Null Byte Check

In IndexBuilder.Add() indexbuilder.go:428-431:

if idx := bytes.IndexByte(doc.Content, 0); idx >= 0 {
    doc.SkipReason = fmt.Sprintf("binary content at byte offset %d", idx)
    doc.Language = "binary"
}

This checks if there's a null byte anywhere in the file content. If found, it:

Sets the SkipReason to indicate where the binary content was detected
Marks the language as "binary"

Then the content is replaced indexbuilder.go:433-440:

if doc.SkipReason != "" {
    doc.Content = []byte(notIndexedMarker + doc.SkipReason)
    doc.Symbols = nil
    doc.SymbolsMetaData = nil
    if doc.Language == "" {
        doc.Language = "skipped"
    }
}

The file content is replaced with a marker string like "NOT-INDEXED: binary content at byte offset 5" indexbuilder.go:386.

Similar Check in DocChecker

The DocChecker.Check() method has an identical check indexbuilder.go:544-546:

if index := bytes.IndexByte(content, 0); index > 0 {
    return fmt.Errorf("binary data at byte offset %d", index)
}

Bottom line: PDFs cannot be indexed in Zoekt because they contain null bytes, which are fundamental to binary file formats. The presence of even a single null byte causes the file to be marked as binary and excluded from text-based indexing. There's no configuration option to override this behavior since Zoekt is designed specifically for text-based source code search.

Langage dominant: TypeScript
Étoiles: 3.9k
Forks: 374
Merge moyen: 21 h 18 min
PR mergées (30 j): 39

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

Lisez l'issue en entier, puis le guide de contribution du projet.
Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
Forkez le dépôt et travaillez sur une branche.
Ouvrez une pull request qui référence le numéro de l'issue.