🐛 Bug Report: nomic-embed-text silently truncates input at 512 tokens, and the reported token count is untruncated

オープン
#15 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
docker, rust

調査の方向性

Start with src/embedding.rs, where init_model sets the model options, and the tokenizer loading and compute_sub_batch paths described in the report. Check tests/embed_e2e.rs for coverage of vector differences and token counts. The report identifies model context lengths but leaves the default cap and memory tradeoff open; done means embedding respects the chosen cap, reports the count actually embedded, and tests cover both behaviors.

索引モデルが issue の本文から書いたものです。

説明

👟 Reproduction steps
  1. Run appwrite/embedding:0.3.1 (the image pinned in appwrite/appwrite's docker-compose.yml) with EMBEDDING_MODELS=nomic-embed-text,all-minilm.
  2. Embed two texts that share a prefix longer than 512 tokens and differ only in the tail, then compare the vectors.
  3. Embed one long text ("word " * 20000) and read tokens in the response.
Script
import json, math, urllib.request

def embed(model, texts):
    req = urllib.request.Request("http://localhost:3000/embed",
        json.dumps({"model": model, "texts": texts}).encode(),
        {"content-type": "application/json"})
    return json.load(urllib.request.urlopen(req))

def cos(a, b):
    return sum(x*y for x, y in zip(a, b)) / math.sqrt(sum(x*x for x in a) * sum(y*y for y in b))

prefix = "fox " * 600   # > 512 tokens
a = embed("nomic-embed-text", [prefix + "cats cats cats cats"])
b = embed("nomic-embed-text", [prefix + "quantum quantum quantum quantum"])
print(cos(a["embeddings"][0], b["embeddings"][0]))   # 1.0, the tails were never seen

print(embed("nomic-embed-text", ["word " * 20000])["tokens"])   # 20002
👍 Expected behavior
  • nomic-embed-text (the default model) embeds up to its real context length (Nomic Embed Text v1.5 supports 8192), or up to a documented, configurable cap.
  • tokens reports the number of tokens the model actually embedded.
👎 Actual Behavior

The model only sees the first 512 tokens and silently ignores the rest. Binary search over the prefix length finds where the tail stops affecting the vector:

Model Tail is ignored once the prefix reaches Cosine of the two vectors
nomic-embed-text 512 tokens 1.000000
all-minilm 256 tokens (its real limit) 1.000000

At the same time tokens counts the full, untruncated input:

Model tokens reported Tokens actually embedded
nomic-embed-text 20002 512
all-minilm 256 256

Impact:

  • Semantic search over anything longer than ~400 English words matches only on the opening. Two long documents with the same intro get identical vectors, with no error or warning.
  • Appwrite's embedding token usage overstates real work by up to ~40x on long inputs.

Root cause:

  1. Truncation at 512. init_model in src/embedding.rs builds InitOptions::new(model) without with_max_length, so fastembed falls back to DEFAULT_MAX_LENGTH = 512 (fastembed-rs/src/text_embedding/mod.rs), and its tokenizer loader applies with_truncation(max_length).
  2. Untruncated count. Tokens are counted with a separate tokenizer from load_tokenizer, which reads tokenizer.json directly with Tokenizer::from_file and never applies that truncation. nomic's tokenizer.json carries none, so every input token is counted. (minilm's does, which is why its count is right.)

compute_sub_batch also assumes "~1000-2000 token code chunks", which suggests more than 512 tokens was the intent.

💡 Proposed fix

I'd like to take this:

  1. Explicit context length per model. Pass .with_max_length(n) in init_model, where n is the model's real context (nomic 8192, gemma 2048, bge-small 512, minilm 256), capped by a new EMBEDDING_MAX_TOKENS env var so self-hosters can trade context for memory.
  2. Count what was embedded. Apply the same TruncationParams { max_length: n } to the counting tokenizer in load_tokenizer, so tokens is what the model saw. Optionally add a truncated count to the response so Appwrite can tell users when input was cut.
  3. Size memory by the real length. Scale mb_per_text in compute_sub_batch with the configured max_length (attention grows with seq²) instead of the fixed 100/40 MB, so a higher cap can't OOM the container.
  4. Tests. In tests/embed_e2e.rs: two texts differing only after 600 tokens must produce different nomic vectors; tokens must never exceed texts.len() * max_length.
  5. Docs. State each model's limit in create-text-embeddings.md in appwrite/appwrite.

Open question: what should the default cap be? 8192 gives nomic its full context but costs much more memory per text. A middle value (for example 2048, in line with the compute_sub_batch comment) covers most real documents. Steps 2 to 4 hold either way.

Related Appwrite endpoint issues: https://github.com/appwrite/appwrite/issues/15532

🎲 Appwrite version

Different version (specify in environment)

💻 Operating system

MacOS

🧱 Your Environment

appwrite/embedding:0.3.1 in Docker on macOS (CPU, ONNX), as pinned by appwrite/appwrite main at c7949c2887 (12 Nov 2026). The relevant code is unchanged on this repo's main (0.3.3): no commit since 0.3.1 touches max_length or the counting tokenizer.

👀 Have you spent some time to check if this issue has been raised before?
  • I checked and didn't find similar issue
🏢 Have you read the Code of Conduct?
主要言語
Rust
スター
0
フォーク
3
平均マージ
10時間 33分
マージ済み PR(30日)
4

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

appwrite/embedding のほかの issue

appwrite/embedding の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。