[Bug]: stale messages in _system/queue/queue.db cause a deterministic SIGSEGV on every startup (no skip/dead-letter path)

Aperta
  • Qualcosa non va?
#6,885 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
18/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, sqlite
Ambito
backend, databases

Direzione di ricerca

The payload names the startup queue database at _system/queue/queue.db and its queue_messages table, but no source file or function where the startup drain runs. Find that drain path first and reproduce the crash by inserting a row with status 'processing' into a copy of the database. Done means startup binds its port even when one row is unreadable, with a logged skip. Expect design questions about dead-lettering, so ask maintainers before choosing an approach.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

kernel

[Bug]: corrupted / stale messages in _system/queue/queue.db cause a deterministic SIGSEGV before the HTTP listener binds

Environment

  • OpenViking: 0.4.23 (also reproduces on 0.5.2.1 with python 3.13 — version-independent)
  • OS: macOS 26.3.1 (arm64)
  • Crash evidence: python SIGSEGV evidenced in DiagnosticReports; stack frames in kevent / tokio-rt-worker threads

Problem

Startup aborts with SIGSEGV before binding any port because the server
drains the startup queue (_system/queue/queue.db) first. If the queue
contains rows its consumer cannot handle (in our case stale
embedding / SessionCommit leftover from a crash window), every start
segmentation-faults identically regardless of:

  • vector index version (isolated the latest version → still crashes)
  • queue WAL (rm queue.db-wal/-shm → still crashes)
  • with_bot: false (→ still crashes)
  • python 3.11 vs 3.13 (→ both crash)
  • OpenViking 0.4.23 vs 0.5.2.1 (→ both crash)

Reproduction

sqlite3 ~/.openviking/data/_system/queue/queue.db \
  "select queue_name, count(*), group_concat(distinct status) from queue_messages group by queue_name;"
# → Embedding/SessionCommit rows with status='processing' left over from an
#    interrupted batch

# Every cold start now segfaults. A single command restores health:
cp queue.db queue.db.full.$(date +%s)
sqlite3 ~/.openviking/data/_system/queue/queue.db "delete from queue_messages;"
openviking-server --config ov.conf   # → immediate healthy start

The delete costs the affected batch's embedding writes (recoverable via
ov reindex), so it is acceptable as an operational workaround — but the
underlying behavior is fragile: one unreadable record takes down the whole
service on every start, with no error message, no skip mechanism, and no
quarantine path.

Suggested fixes

  1. Defensive drain: wrap per-message queue consumption in a try/except that
    logs, skips (or moves to a dead-letter table), and continues — a bad record
    must never prevent the server from binding its ports.
  2. If a message fails N times or fails to deserialize, quarantine it into a
    queue_dead_letter table with the job payload for later inspection.
  3. Surface a startup warning ("skipped X unreadable queue messages, inspect
    SKIPPED_QUEUE_MESSAGES.md") instead of a silent process death.

Compounding context

This was observed on 2026-11-01 while investigating a day of intermittent
crashes; the orphaned-gateway port issue (filed separately) amplified each
occurrence into a 15–30 min outage. The two defects interact but are
independent.

Lingua principale
Python
Stelle
41.8k
Fork
3.4k
Merge medio
19h 29m
PR unite (30g)
500

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di volcengine/OpenViking

Tutte le issue di volcengine/OpenViking

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.