🐛 Bug Report: nomic-embed-text silently truncates input at 512 tokens, and the reported token count is untruncated
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 45/100
- issue の種類
- バグ
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- docker, rust
調査の方向性
Start with src/embedding.rs, where init_model sets the model options, and the tokenizer loading and compute_sub_batch paths described in the report. Check tests/embed_e2e.rs for coverage of vector differences and token counts. The report identifies model context lengths but leaves the default cap and memory tradeoff open; done means embedding respects the chosen cap, reports the count actually embedded, and tests cover both behaviors.
索引モデルが issue の本文から書いたものです。
説明
👟 Reproduction steps
- Run
appwrite/embedding:0.3.1(the image pinned in appwrite/appwrite'sdocker-compose.yml) withEMBEDDING_MODELS=nomic-embed-text,all-minilm. - Embed two texts that share a prefix longer than 512 tokens and differ only in the tail, then compare the vectors.
- Embed one long text (
"word " * 20000) and readtokensin the response.
Script
import json, math, urllib.request
def embed(model, texts):
req = urllib.request.Request("http://localhost:3000/embed",
json.dumps({"model": model, "texts": texts}).encode(),
{"content-type": "application/json"})
return json.load(urllib.request.urlopen(req))
def cos(a, b):
return sum(x*y for x, y in zip(a, b)) / math.sqrt(sum(x*x for x in a) * sum(y*y for y in b))
prefix = "fox " * 600 # > 512 tokens
a = embed("nomic-embed-text", [prefix + "cats cats cats cats"])
b = embed("nomic-embed-text", [prefix + "quantum quantum quantum quantum"])
print(cos(a["embeddings"][0], b["embeddings"][0])) # 1.0, the tails were never seen
print(embed("nomic-embed-text", ["word " * 20000])["tokens"]) # 20002
👍 Expected behavior
nomic-embed-text(the default model) embeds up to its real context length (Nomic Embed Text v1.5 supports 8192), or up to a documented, configurable cap.tokensreports the number of tokens the model actually embedded.
👎 Actual Behavior
The model only sees the first 512 tokens and silently ignores the rest. Binary search over the prefix length finds where the tail stops affecting the vector:
| Model | Tail is ignored once the prefix reaches | Cosine of the two vectors |
|---|---|---|
| nomic-embed-text | 512 tokens | 1.000000 |
| all-minilm | 256 tokens (its real limit) | 1.000000 |
At the same time tokens counts the full, untruncated input:
| Model | tokens reported |
Tokens actually embedded |
|---|---|---|
| nomic-embed-text | 20002 | 512 |
| all-minilm | 256 | 256 |
Impact:
- Semantic search over anything longer than ~400 English words matches only on the opening. Two long documents with the same intro get identical vectors, with no error or warning.
- Appwrite's embedding token usage overstates real work by up to ~40x on long inputs.
Root cause:
- Truncation at 512.
init_modelinsrc/embedding.rsbuildsInitOptions::new(model)withoutwith_max_length, so fastembed falls back toDEFAULT_MAX_LENGTH = 512(fastembed-rs/src/text_embedding/mod.rs), and its tokenizer loader applieswith_truncation(max_length). - Untruncated count. Tokens are counted with a separate tokenizer from
load_tokenizer, which readstokenizer.jsondirectly withTokenizer::from_fileand never applies that truncation. nomic'stokenizer.jsoncarries none, so every input token is counted. (minilm's does, which is why its count is right.)
compute_sub_batch also assumes "~1000-2000 token code chunks", which suggests more than 512 tokens was the intent.
💡 Proposed fix
I'd like to take this:
- Explicit context length per model. Pass
.with_max_length(n)ininit_model, wherenis the model's real context (nomic 8192, gemma 2048, bge-small 512, minilm 256), capped by a newEMBEDDING_MAX_TOKENSenv var so self-hosters can trade context for memory. - Count what was embedded. Apply the same
TruncationParams { max_length: n }to the counting tokenizer inload_tokenizer, sotokensis what the model saw. Optionally add atruncatedcount to the response so Appwrite can tell users when input was cut. - Size memory by the real length. Scale
mb_per_textincompute_sub_batchwith the configuredmax_length(attention grows with seq²) instead of the fixed 100/40 MB, so a higher cap can't OOM the container. - Tests. In
tests/embed_e2e.rs: two texts differing only after 600 tokens must produce different nomic vectors;tokensmust never exceedtexts.len() * max_length. - Docs. State each model's limit in
create-text-embeddings.mdin appwrite/appwrite.
Open question: what should the default cap be? 8192 gives nomic its full context but costs much more memory per text. A middle value (for example 2048, in line with the compute_sub_batch comment) covers most real documents. Steps 2 to 4 hold either way.
Related Appwrite endpoint issues: https://github.com/appwrite/appwrite/issues/15532
🎲 Appwrite version
Different version (specify in environment)
💻 Operating system
MacOS
🧱 Your Environment
appwrite/embedding:0.3.1 in Docker on macOS (CPU, ONNX), as pinned by appwrite/appwrite main at c7949c2887 (12 Nov 2026). The relevant code is unchanged on this repo's main (0.3.3): no commit since 0.3.1 touches max_length or the counting tokenizer.
👀 Have you spent some time to check if this issue has been raised before?
- I checked and didn't find similar issue
🏢 Have you read the Code of Conduct?
- I have read the Code of Conduct
- 主要言語
- Rust
- スター
- 0
- フォーク
- 3
- 平均マージ
- 10時間 33分
- マージ済み PR(30日)
- 4
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
appwrite/embedding のほかの issue
-
Dependency Dashboardオープン
難易度 4/5 3〜5日 初心者へのやさしさ 15/100
appwrite/embedding の issue をすべて見る
似ている issue
-
marketing
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
flashbots/panetiere#46 · コメント 1 件 ·
メンテナーはふだん 5 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
jlcodes99/cockpit-tools#3190 ·
メンテナーはふだん 4 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
Feyisara2108/stellargive#1376 ·
メンテナーはふだん 1 日以内に返信
-
area:post bug effort:medium P7 source:agent
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信