[Bug]: Retrieval rerank fallback statistics stay zero after provider failure

Aperta Adatta ai principianti
  • Qualcosa non va?
#7,014 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@97Three ci sta già lavorando.

Dal 4/11/2026.

  • #7015 di @97Three — aperta
  • #7043 di @zuizhuai — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
78/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, observability

Direzione di ricerca

Inizia da HierarchicalRetriever.retrieve() in openviking/retrieve/hierarchical_retriever.py e verifica come l’esito del rerank viene passato a record_query(). L’issue indica RerankScores.reranked come segnale da esaminare e tests/retrieve/test_hierarchical_retriever_rerank.py come punto in cui aggiungere la copertura di regressione. Il lavoro è concluso quando la riproduzione dell’interruzione registra un fallback preservando il punteggio vettoriale e i casi indicati QUICK, non configurato e senza candidati non incrementano il fallback.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug kernel
Issue Origin

Observed or reproduced in a real environment

Bug Description

When the reranker fails, HierarchicalRetriever.retrieve correctly falls back to vector scores, but records rerank_used=True and never passes rerank_fallback to record_query(). Consequently RetrievalObserver and retrieval metrics report zero fallbacks even during an outage. Reproduced locally through retrieve() with deterministic fake storage/embedder and a rerank client that raises; no live provider or deployment incident is claimed.

Steps to Reproduce
  1. On main ef6f8f91085ddd0b7264f2bbd1205d3e37370e4a, reset get_stats_collector().
  2. Configure a reranker, provide one vector candidate with score 0.8 and a nonempty abstract, and make rerank_batch raise RuntimeError.
  3. Call the real HierarchicalRetriever.retrieve(), then inspect the collector snapshot.
  4. The result retains score 0.8, but rerank_used=1 and rerank_fallback=0. The example below reuses existing repository test doubles.
Expected Behavior

Record fallback when a rerank-enabled retrieval retains vector scores because the reranker failed or returned unusable scores. Proposed semantics: rerank_used indicates at least one model-produced score; rerank_fallback indicates at least one candidate retained its vector score. A partially reranked query may increment both counters. QUICK queries, unconfigured rerankers and queries with no candidates should not increment fallback.

Actual Behavior

Observed after the synthetic provider outage: returned_score=0.8, rerank_used=1, rerank_fallback=0. The existing RerankScores.reranked flags already identify per-candidate fallback, but retrieve() does not use them when recording aggregate statistics.

Minimal Reproducible Example
import asyncio
from unittest.mock import patch
from openviking.retrieve.hierarchical_retriever import HierarchicalRetriever
from openviking.retrieve.retrieval_stats import get_stats_collector
from tests.retrieve.test_hierarchical_retriever_rerank import (
    DummyStorage, DummyEmbedder, _result, _ctx, _query, _config,
)

class BrokenReranker:
    def rerank_batch(self, query, documents):
        raise RuntimeError("simulated provider outage")

async def main():
    stats = get_stats_collector()
    stats.reset()
    with patch("openviking.retrieve.hierarchical_retriever.RerankClient.from_config", return_value=BrokenReranker()):
        retriever = HierarchicalRetriever(
            DummyStorage([_result("viking://resources/doc.md", 0.8)]),
            DummyEmbedder(), _config(),
        )
        result = await retriever.retrieve(_query(), ctx=_ctx())
        snapshot = stats.snapshot()
        print(result.matched_contexts[0].score, snapshot.rerank_used, snapshot.rerank_fallback)
        # actual: 0.8 1 0

asyncio.run(main())
Error Logs
OpenViking Version

main @ ef6f8f91085ddd0b7264f2bbd1205d3e37370e4a

Python Version

3.11.8

Operating System

Windows

Model Backend

None

Additional Context

I will work on a focused fix and regression tests in the existing retrieval test module. The change will only report the actual rerank outcome; it should not change ranking, thresholds, fallback scores or result content. No duplicate issue or open PR addressing this counter propagation was found in the search.

Lingua principale
Python
Stelle
41.3k
Fork
3.4k
Merge medio
1g 6h
PR unite (30g)
454

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di volcengine/OpenViking

Tutte le issue di volcengine/OpenViking

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.