🐛 Bug Report: nomic-embed-text silently truncates input at 512 tokens, and the reported token count is untruncated

Ouverte
#15 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
45/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
docker, rust

Piste de recherche

Start with src/embedding.rs, where init_model sets the model options, and the tokenizer loading and compute_sub_batch paths described in the report. Check tests/embed_e2e.rs for coverage of vector differences and token counts. The report identifies model context lengths but leaves the default cap and memory tradeoff open; done means embedding respects the chosen cap, reports the count actually embedded, and tests cover both behaviors.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

👟 Reproduction steps
  1. Run appwrite/embedding:0.3.1 (the image pinned in appwrite/appwrite's docker-compose.yml) with EMBEDDING_MODELS=nomic-embed-text,all-minilm.
  2. Embed two texts that share a prefix longer than 512 tokens and differ only in the tail, then compare the vectors.
  3. Embed one long text ("word " * 20000) and read tokens in the response.
Script
import json, math, urllib.request

def embed(model, texts):
    req = urllib.request.Request("http://localhost:3000/embed",
        json.dumps({"model": model, "texts": texts}).encode(),
        {"content-type": "application/json"})
    return json.load(urllib.request.urlopen(req))

def cos(a, b):
    return sum(x*y for x, y in zip(a, b)) / math.sqrt(sum(x*x for x in a) * sum(y*y for y in b))

prefix = "fox " * 600   # > 512 tokens
a = embed("nomic-embed-text", [prefix + "cats cats cats cats"])
b = embed("nomic-embed-text", [prefix + "quantum quantum quantum quantum"])
print(cos(a["embeddings"][0], b["embeddings"][0]))   # 1.0, the tails were never seen

print(embed("nomic-embed-text", ["word " * 20000])["tokens"])   # 20002
👍 Expected behavior
  • nomic-embed-text (the default model) embeds up to its real context length (Nomic Embed Text v1.5 supports 8192), or up to a documented, configurable cap.
  • tokens reports the number of tokens the model actually embedded.
👎 Actual Behavior

The model only sees the first 512 tokens and silently ignores the rest. Binary search over the prefix length finds where the tail stops affecting the vector:

Model Tail is ignored once the prefix reaches Cosine of the two vectors
nomic-embed-text 512 tokens 1.000000
all-minilm 256 tokens (its real limit) 1.000000

At the same time tokens counts the full, untruncated input:

Model tokens reported Tokens actually embedded
nomic-embed-text 20002 512
all-minilm 256 256

Impact:

  • Semantic search over anything longer than ~400 English words matches only on the opening. Two long documents with the same intro get identical vectors, with no error or warning.
  • Appwrite's embedding token usage overstates real work by up to ~40x on long inputs.

Root cause:

  1. Truncation at 512. init_model in src/embedding.rs builds InitOptions::new(model) without with_max_length, so fastembed falls back to DEFAULT_MAX_LENGTH = 512 (fastembed-rs/src/text_embedding/mod.rs), and its tokenizer loader applies with_truncation(max_length).
  2. Untruncated count. Tokens are counted with a separate tokenizer from load_tokenizer, which reads tokenizer.json directly with Tokenizer::from_file and never applies that truncation. nomic's tokenizer.json carries none, so every input token is counted. (minilm's does, which is why its count is right.)

compute_sub_batch also assumes "~1000-2000 token code chunks", which suggests more than 512 tokens was the intent.

💡 Proposed fix

I'd like to take this:

  1. Explicit context length per model. Pass .with_max_length(n) in init_model, where n is the model's real context (nomic 8192, gemma 2048, bge-small 512, minilm 256), capped by a new EMBEDDING_MAX_TOKENS env var so self-hosters can trade context for memory.
  2. Count what was embedded. Apply the same TruncationParams { max_length: n } to the counting tokenizer in load_tokenizer, so tokens is what the model saw. Optionally add a truncated count to the response so Appwrite can tell users when input was cut.
  3. Size memory by the real length. Scale mb_per_text in compute_sub_batch with the configured max_length (attention grows with seq²) instead of the fixed 100/40 MB, so a higher cap can't OOM the container.
  4. Tests. In tests/embed_e2e.rs: two texts differing only after 600 tokens must produce different nomic vectors; tokens must never exceed texts.len() * max_length.
  5. Docs. State each model's limit in create-text-embeddings.md in appwrite/appwrite.

Open question: what should the default cap be? 8192 gives nomic its full context but costs much more memory per text. A middle value (for example 2048, in line with the compute_sub_batch comment) covers most real documents. Steps 2 to 4 hold either way.

Related Appwrite endpoint issues: https://github.com/appwrite/appwrite/issues/15532

🎲 Appwrite version

Different version (specify in environment)

💻 Operating system

MacOS

🧱 Your Environment

appwrite/embedding:0.3.1 in Docker on macOS (CPU, ONNX), as pinned by appwrite/appwrite main at c7949c2887 (12 Nov 2026). The relevant code is unchanged on this repo's main (0.3.3): no commit since 0.3.1 touches max_length or the counting tokenizer.

👀 Have you spent some time to check if this issue has been raised before?
  • I checked and didn't find similar issue
🏢 Have you read the Code of Conduct?
Langage dominant
Rust
Étoiles
0
Forks
3
Merge moyen
10 h 33 min
PR mergées (30 j)
4

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de appwrite/embedding

Toutes les issues de appwrite/embedding

Issues similaires

Plus d'issues Rust

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.