[Bug]: Revision-pinned HF downloads can falsely report intact files as corrupted

Aperta
  • Qualcosa non va?
#870 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@N7766 ci sta già lavorando.

Dal 7/11/2026.

  • #871 di @N7766 — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
82/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start with ModelManagement.download_files_from_huggingface in fastembed/common/model_management.py and trace how the revision is passed to model_info, list_repo_tree, and snapshot_download. Use the offline reproduction and the four regression cases named in the issue to check release tags, explicit SHAs, and moving refs. Done means metadata and downloaded files resolve to the same commit and the targeted offline tests pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

[Bug]: Revision-pinned HF downloads can falsely report intact files as corrupted

What happened?

Reproduced at f91d9168c0291daeafb0a54cebc7c5b4b83f312f (FastEmbed 0.9.1): ModelManagement.download_files_from_huggingface(..., revision="release-v1") passes the existing revision option to snapshot_download but not to the preceding model_info request. Metadata therefore describes main while the downloaded files describe the requested release. A file-size change makes the real local integrity check reject an intact pinned snapshot.

The same consistency problem can happen when a default/named branch moves between metadata resolution and download. I reproduced that race with simulated responses; I am not claiming to have observed a live repository move during a download.

This concerns the online path. #798 addresses a different offline-cache validation path, and the merged #780 limits metadata collection to the downloaded snapshot but still obtains metadata from main.

Expected behavior

Use the requested revision for metadata, then download the exact resolved commit so metadata and files belong to the same snapshot.

Offline reproduction

The three external Hub responses below are mocked. Tiny local file writes, metadata collection, and integrity checks are real. No model download, inference, GPU, or credentials are required beyond installing FastEmbed's normal runtime dependencies.

"""Offline reproduction: Hub responses are simulated; local file verification is real."""

from pathlib import Path
from tempfile import TemporaryDirectory
from types import SimpleNamespace
from unittest.mock import patch

from huggingface_hub.hf_api import RepoFile

from fastembed.common import model_management
from fastembed.common.model_management import ModelManagement


OLD, MAIN = "a" * 40, "b" * 40
FILES = {OLD: b"old model fixture", MAIN: b"new model fixture with a different size"}


with TemporaryDirectory() as cache:
    def model_info(repo_id, revision=None, **kwargs):
        commit = OLD if revision in ("release-v1", OLD) else MAIN
        print(f"model_info: revision={revision!r}, commit={commit}")
        return SimpleNamespace(sha=commit)

    def list_repo_tree(repo_id, revision, **kwargs):
        print(f"list_repo_tree: commit={revision}")
        return [RepoFile(path="model.onnx", size=len(FILES[revision]), oid=revision)]

    def snapshot_download(revision=None, **kwargs):
        commit = OLD if revision in ("release-v1", OLD) else MAIN
        print(f"snapshot_download: revision={revision!r}, commit={commit}")
        model_file = Path(cache) / "models--qdrant--fixture" / "snapshots" / commit / "model.onnx"
        model_file.parent.mkdir(parents=True)
        model_file.write_bytes(FILES[commit])
        return str(model_file.parent)

    with (
        patch.object(model_management, "model_info", model_info),
        patch.object(model_management, "list_repo_tree", list_repo_tree),
        patch.object(model_management, "snapshot_download", snapshot_download),
    ):
        try:
            result = ModelManagement.download_files_from_huggingface(
                "qdrant/fixture",
                cache_dir=cache,
                extra_patterns=["model.onnx"],
                revision="release-v1",
            )
            print("Valid pinned snapshot accepted:", Path(result).name == OLD)
        except ValueError as exc:
            print(type(exc).__name__ + ":", exc)

On the unpatched source:

model_info: revision=None, commit=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
list_repo_tree: commit=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
snapshot_download: revision='release-v1', commit=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
ValueError: Files have been corrupted during downloading process. Please check your internet connection and try again.
Validation and environment

Python 3.11.13 / macOS arm64, FastEmbed source main at the commit above, huggingface-hub 2.2.0, pytest 9.1.1. Four new offline regressions (release tag, explicit SHA, default ref movement, named ref movement) all fail on the base with this false corruption error. Resolving the requested revision and pinning the download to its returned SHA makes them pass; the targeted cache/common suite passed all 15 cases with HF_HUB_OFFLINE=1. Adding the existing tokenizer-fixture suite under offline/empty-cache conditions produces 39 setup errors because its fixture initializes BAAI/bge-small-en-v1.5; it is excluded from this offline acceptance set. Full model-download/inference and platform-matrix tests were not run.

Prepared with OpenAI Codex assistance; the reproduction and stated checks were executed locally.

Lingua principale
Python
Stelle
3.3k
Fork
296
Merge medio
4g 11h
PR unite (30g)
38

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di qdrant/fastembed

Tutte le issue di qdrant/fastembed

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.