🐛 Bug Report: nomic-embed-text silently truncates input at 512 tokens, and the reported token count is untruncated

Đang mở
#15 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
45/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
docker, rust
Lĩnh vực
backend, machine-learning

Hướng nghiên cứu

Start with src/embedding.rs, where init_model sets the model options, and the tokenizer loading and compute_sub_batch paths described in the report. Check tests/embed_e2e.rs for coverage of vector differences and token counts. The report identifies model context lengths but leaves the default cap and memory tradeoff open; done means embedding respects the chosen cap, reports the count actually embedded, and tests cover both behaviors.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

👟 Reproduction steps
  1. Run appwrite/embedding:0.3.1 (the image pinned in appwrite/appwrite's docker-compose.yml) with EMBEDDING_MODELS=nomic-embed-text,all-minilm.
  2. Embed two texts that share a prefix longer than 512 tokens and differ only in the tail, then compare the vectors.
  3. Embed one long text ("word " * 20000) and read tokens in the response.
Script
import json, math, urllib.request

def embed(model, texts):
    req = urllib.request.Request("http://localhost:3000/embed",
        json.dumps({"model": model, "texts": texts}).encode(),
        {"content-type": "application/json"})
    return json.load(urllib.request.urlopen(req))

def cos(a, b):
    return sum(x*y for x, y in zip(a, b)) / math.sqrt(sum(x*x for x in a) * sum(y*y for y in b))

prefix = "fox " * 600   # > 512 tokens
a = embed("nomic-embed-text", [prefix + "cats cats cats cats"])
b = embed("nomic-embed-text", [prefix + "quantum quantum quantum quantum"])
print(cos(a["embeddings"][0], b["embeddings"][0]))   # 1.0, the tails were never seen

print(embed("nomic-embed-text", ["word " * 20000])["tokens"])   # 20002
👍 Expected behavior
  • nomic-embed-text (the default model) embeds up to its real context length (Nomic Embed Text v1.5 supports 8192), or up to a documented, configurable cap.
  • tokens reports the number of tokens the model actually embedded.
👎 Actual Behavior

The model only sees the first 512 tokens and silently ignores the rest. Binary search over the prefix length finds where the tail stops affecting the vector:

Model Tail is ignored once the prefix reaches Cosine of the two vectors
nomic-embed-text 512 tokens 1.000000
all-minilm 256 tokens (its real limit) 1.000000

At the same time tokens counts the full, untruncated input:

Model tokens reported Tokens actually embedded
nomic-embed-text 20002 512
all-minilm 256 256

Impact:

  • Semantic search over anything longer than ~400 English words matches only on the opening. Two long documents with the same intro get identical vectors, with no error or warning.
  • Appwrite's embedding token usage overstates real work by up to ~40x on long inputs.

Root cause:

  1. Truncation at 512. init_model in src/embedding.rs builds InitOptions::new(model) without with_max_length, so fastembed falls back to DEFAULT_MAX_LENGTH = 512 (fastembed-rs/src/text_embedding/mod.rs), and its tokenizer loader applies with_truncation(max_length).
  2. Untruncated count. Tokens are counted with a separate tokenizer from load_tokenizer, which reads tokenizer.json directly with Tokenizer::from_file and never applies that truncation. nomic's tokenizer.json carries none, so every input token is counted. (minilm's does, which is why its count is right.)

compute_sub_batch also assumes "~1000-2000 token code chunks", which suggests more than 512 tokens was the intent.

💡 Proposed fix

I'd like to take this:

  1. Explicit context length per model. Pass .with_max_length(n) in init_model, where n is the model's real context (nomic 8192, gemma 2048, bge-small 512, minilm 256), capped by a new EMBEDDING_MAX_TOKENS env var so self-hosters can trade context for memory.
  2. Count what was embedded. Apply the same TruncationParams { max_length: n } to the counting tokenizer in load_tokenizer, so tokens is what the model saw. Optionally add a truncated count to the response so Appwrite can tell users when input was cut.
  3. Size memory by the real length. Scale mb_per_text in compute_sub_batch with the configured max_length (attention grows with seq²) instead of the fixed 100/40 MB, so a higher cap can't OOM the container.
  4. Tests. In tests/embed_e2e.rs: two texts differing only after 600 tokens must produce different nomic vectors; tokens must never exceed texts.len() * max_length.
  5. Docs. State each model's limit in create-text-embeddings.md in appwrite/appwrite.

Open question: what should the default cap be? 8192 gives nomic its full context but costs much more memory per text. A middle value (for example 2048, in line with the compute_sub_batch comment) covers most real documents. Steps 2 to 4 hold either way.

Related Appwrite endpoint issues: https://github.com/appwrite/appwrite/issues/15532

🎲 Appwrite version

Different version (specify in environment)

💻 Operating system

MacOS

🧱 Your Environment

appwrite/embedding:0.3.1 in Docker on macOS (CPU, ONNX), as pinned by appwrite/appwrite main at c7949c2887 (12 Nov 2026). The relevant code is unchanged on this repo's main (0.3.3): no commit since 0.3.1 touches max_length or the counting tokenizer.

👀 Have you spent some time to check if this issue has been raised before?
  • I checked and didn't find similar issue
🏢 Have you read the Code of Conduct?
Ngôn ngữ chính
Rust
Star
0
Fork
3
Merge trung bình
10 giờ 33 phút
Pull request đã merge (30 ngày)
4

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của appwrite/embedding

Tất cả issue của appwrite/embedding

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.