Late interaction normalization produces nonfinite float16 embeddings

Chiusa
  • Qualcosa non va?
#825 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@IlyaKhalafi ci sta già lavorando.

Dal 15/10/2026.

  • #826 di @IlyaKhalafi — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
Mezza giornata
Idoneità per principianti
70/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
numpy, python
Ambito
machine-learning

Direzione di ricerca

Start with _post_process_onnx_output in the Colbert and LateOn late-interaction postprocessors, which compute L2 norms in float16. Run the reproduction from the issue, which builds an OnnxOutputContext directly, and check that the output has infinities or NaNs. Read the shared normalization helper that the issue says already handles float16 safely, and reuse its approach in both postprocessors. Done when float16 output keeps its dtype, the norms come out near [1, 0, 1], and existing padding, punctuation and query tests still pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Colbert._post_process_onnx_output and LateOn._post_process_onnx_output compute L2 norms in float16 when an ONNX model returns float16 embeddings. Small finite embeddings underflow to a zero norm, producing infinities. Zero vectors produce NaNs because the 1e-12 denominator floor also becomes zero in float16. Larger finite values can overflow the norm and become zero vectors.

I reproduced this on main at de502ca8 with NumPy 2.5.3 on Linux. The shared normalization helper already handles float16 safely, but these two postprocessors use their own normalization.

This isolates postprocessing without loading model weights.

import numpy as np
from fastembed.common.onnx_model import OnnxOutputContext
from fastembed.late_interaction.colbert import Colbert

model = object.__new__(Colbert)
model._ensure_tokenizer = lambda: None
model.skip_list = set()
model.pad_token_id = 0

vectors = np.full((1, 3, 128), 1e-5, dtype=np.float16)
vectors[0, 1] = 0
vectors[0, 2] = 100
context = OnnxOutputContext(
    model_output=vectors,
    attention_mask=np.ones((1, 3), dtype=np.int64),
    input_ids=np.array([[1, 2, 3]], dtype=np.int64),
)
result = next(iter(model._post_process_onnx_output(context)))
print(np.isfinite(result).all())
print(np.linalg.norm(result.astype(np.float32), axis=1))

The output contains infinities and NaNs, and the large vector has norm zero. Expected output is finite with norms near [1, 0, 1]. The same reproduction works with LateOn.

Float16 output should retain its dtype and be normalized using enough precision to avoid these errors. Existing padding, punctuation filtering and query behavior should be preserved.

Lingua principale
Python
Stelle
3.3k
Fork
296
Merge medio
4g 3h
PR unite (30g)
41

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di qdrant/fastembed

Tutte le issue di qdrant/fastembed

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.