Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Watchtower /health and /search share a 1-hour-TTL index handle, serving a replaced inode after an out-of-process reindex

Aperta
  • Qualcosa non va?
#50 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, search

Direzione di ricerca

Start with web/embeddings.py at STALE_SECONDS and _get_db(), then trace how the index handle is opened and used by /search and /health in web/app.py. Check how reindex replaces the index file and whether its device/inode identity is available for comparison. Done when an out-of-process reindex is detected without waiting for the TTL and search and health use the refreshed index.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Watchtower's web process holds a long-lived index handle that is only reopened after a
one-hour TTL, not when the index file underneath it is replaced. Any reindex performed
by a different process — the hourly cron job, a manual fw index reindex — swaps the
file via os.replace() while the web process keeps its handle on the old inode.

Code path
  • web/embeddings.py:111 — STALE_SECONDS = 3600
  • web/embeddings.py:465-494 — _get_db() reopens only once that TTL has elapsed
  • web/app.py:349-360 — /health's chunks field is read from that same process-global
    _db handle
  • web/embeddings.py:754, 1246, 1384, 1501 — /search and /recall use the same handle
Consequence

/search serves results from the replaced inode for up to an hour after any reindex
the web process did not itself trigger. Newly indexed content is invisible to search for
that window, and nothing surfaces it as stale.

Worse for diagnosis: /health reports chunks from the stale handle while
index_age_seconds on the same JSON line comes from a fresher manifest read. So the health
endpoint can show a recent index age beside a chunk count from an hour-old inode — the one
surface an operator would check to detect this shows a self-consistent-looking pair of
numbers sourced from two different points in time.

This was originally observed downstream as a count mismatch (/health reporting 13242
chunks against a completed reindex that wrote 13346). The observation asked whether
/search was also affected and left it open. Reading the code answers it: yes, the same
handle, so yes.

Note this is not fixed by scheduling the reindex correctly — the downstream task that
installed the hourly job never touched _get_db's TTL, so the file refreshes hourly and
the server does not.

Suggested fix

Invalidate on inode identity rather than elapsed time: stat the index path and reopen when
st_ino/st_dev differ from the open handle's, in addition to the TTL. Cheap, and it
makes the reindex swap self-announcing instead of requiring the reader to wait it out.

Filed from project 1409-sprind under its observation-inbox triage task, confirmed by
code reading on 1.6.769 rather than by reproducing the timing window.

Lingua principale
Shell
Stelle
16
Fork
6
Merge medio
6h 24m
PR unite (30g)
5

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di DimitriGeelen/agentic-engineering-framework

Tutte le issue di DimitriGeelen/agentic-engineering-framework

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.