Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

[bug/rfe] Rebuild or mark repos stale when zoekt shard files are missing but DB marks repos indexed

Offen
  • Stimmt etwas nicht?
#1,210 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@reddynitish arbeitet bereits daran.

Seit 30.8.2026.

  • #1304 von @rudraasakariya — ohne Merge geschlossen
  • #1350 von @RitwijParmar — ohne Merge geschlossen
  • #1621 von @reddynitish — ohne Merge geschlossen

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
58/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
typescript
Bereich
backend, search

Rechercherichtung

Start in packages/backend/src/repoIndexManager.ts with scheduleIndexJobs() and cleanupOrphanedDiskResources(), then inspect startup directory handling in packages/backend/src/index.ts and shard creation in packages/backend/src/zoekt.ts. Reproduce by deleting the shard files and restarting with a long reindex interval; done means missing shards trigger recovery and search returns the repository contents again.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

bug triage needed

Summary

If the zoekt index directory is deleted while the Sourcebot database and repo clones remain intact, Sourcebot starts successfully but does not automatically rebuild the missing shards. Repos remain marked as indexed in the DB, while zoekt has no shard files to search, so search returns incomplete or empty results from an otherwise healthy instance.

This matters for Kubernetes deployments that want to place only .sourcebot/index on more performant ephemeral/local storage while keeping DB/repos on durable storage.

Version

Observed on:

  • Sourcebot image: ghcr.io/sourcebot-dev/sourcebot:v4.17.1

Background / Use Case

For performance reasons, we are evaluating a hybrid storage layout:

  • durable PVC for /data
  • node-local / ephemeral storage for /data/.sourcebot/index

The zoekt index is derived/rebuildable, while the DB and repo clones should remain durable. This can improve cold/random read performance for large zoekt indexes, but pod replacement would delete only the index directory.

In that scenario, Sourcebot should ideally detect that indexed repos no longer have shard files and enqueue rebuilds, or otherwise mark those repos as not indexed.

Current Behavior

When /data/.sourcebot/index is empty but the DB still has repos with fresh indexedAt values:

  • Sourcebot starts normally.
  • zoekt tolerates the empty index directory.
  • The repo remains marked as indexed/completed in the DB.
  • No new indexing job is scheduled.
  • Search returns zero/incomplete results.
  • Manual reindex through /api/index-repo successfully rebuilds the missing shard.

Expected Behavior

On startup, or during repo-index scheduling, Sourcebot should detect repos that are marked indexed in the DB but have no corresponding zoekt shard files.

Possible expected behaviors:

  • enqueue index jobs for affected repos,
  • set indexedAt = null for affected repos so the existing scheduler picks them up,
  • or expose a supported startup/recovery command that rebuilds all missing index shards.

Relevant Code Paths

From a quick read of the code:

  • packages/backend/src/constants.ts

    • REPOS_CACHE_DIR = DATA_CACHE_DIR/repos
    • INDEX_CACHE_DIR = DATA_CACHE_DIR/index

  • packages/backend/src/index.ts

    • creates the repos and index directories at startup if absent

  • packages/backend/src/repoIndexManager.ts

    • scheduleIndexJobs() selects repos where indexedAt is null or older than reindexIntervalMs
    • cleanupOrphanedDiskResources() removes shard files with no corresponding DB repo, but does not detect the inverse: DB repo exists and is marked indexed, but shard files are missing

  • packages/backend/src/zoekt.ts

    • indexGitRepository() invokes zoekt-git-index -index ${INDEX_CACHE_DIR}

Suggested Fix

Add startup or scheduler reconciliation that compares DB repo state with shard files in INDEX_CACHE_DIR.

For each repo with indexedAt != null, verify that at least one shard exists with that repo's shard prefix. If no shard exists, either:

  1. enqueue an index job for that repo, or
  2. mark the repo as stale by clearing indexedAt, allowing the existing scheduler to enqueue it.

This would make derived/ephemeral index storage safe to use with durable DB/repos.

To reproduce

Reproduction

Use a small Sourcebot instance with a local file:// repo.

Example config:

{
  "connections": {
    "local-repos": {
      "type": "git",
      "url": "file:///data/local-repos/*"
    }
  },
  "settings": {
    "reindexIntervalMs": 86400000,
    "reindexRepoPollingIntervalMs": 1000,
    "maxRepoIndexingJobConcurrency": 1
  }
}

The long reindexIntervalMs is intentional so we can distinguish immediate missing-shard recovery from the normal periodic reindex.

Steps:

  1. Create a local git repo under /data/local-repos/testrepo.
  2. Start Sourcebot.
  3. Wait for the repo to index successfully.
  4. Confirm:
    • a shard exists under /data/.sourcebot/index
    • the repo has indexedAt set
    • search returns a result from the repo
  5. Delete only the zoekt shard files:
rm -f /data/.sourcebot/index/*.zoekt \
      /data/.sourcebot/index/*.zoekt.meta \
      /data/.sourcebot/index/*.tmp
  1. Restart Sourcebot with the same DB and repo data.
  2. Observe that no new index job is created and search returns no results.

Manual recovery works:

curl -s -X POST http://localhost:3060/api/index-repo \
  -H "Content-Type: application/json" \
  -d '{"repoId":1}'

After this, the shard is rebuilt and search works again.

Sourcebot deployment information

ghcr.io/sourcebot-dev/sourcebot:v4.17.1

Additional information

No response

Vorherrschende Sprache
TypeScript
Sterne
3.9k
Forks
374
Ø Merge
2 T. 21 Std.
Gemergte PRs (30 T.)
43

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus sourcebot-dev/sourcebot

Alle Issues in sourcebot-dev/sourcebot

Ähnliche Issues

Weitere Issues zu TypeScript

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.