Embedding document fails if tokens length > n_ctx

エラーはllama_cpp/llama.py、具体的には549行目のevalメソッドで発生します。まず、トークンの扱いを理解するために、create_embeddingメソッド（869行目）とembedメソッド（905行目）を調べてください。修正には、evalを呼び出す前にトークン長がn_ctxを超えていないか確認し、明確なエラーまたは警告を発生させる処理を追加することが含まれる可能性があります。コードベース内の既存のバリデーションパターンを探し、長いドキュメントでテストして問題を再現してください。

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [X ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ X] I carefully followed the README.md.
• [ X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [ X] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Embedding a document, if embedding document tokens are > to n_ctx , it should warn or exit()

Current Behavior

it fails with error
File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in embed_documents
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 905, in embed
return list(map(float, self.create_embedding(input)["data"][0]["embedding"]))
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 869, in create_embedding
self.eval(tokens)
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 549, in eval
self.input_ids[self.n_tokens : self.n_tokens + n_tokens] = batch
ValueError: could not broadcast input array from shape (8,) into shape (0,)

Environment and Context

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

document = open("DOC/mybigdocument.txt,"r")
text = document.read()
embeddings = LlamaCppEmbeddings(model_path=model_path,verbose=False)
embeddings.embed_documents([text])

Failure Logs

Pl File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in embed_documents
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 905, in embed
return list(map(float, self.create_embedding(input)["data"][0]["embedding"]))
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 869, in create_embedding
self.eval(tokens)
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 549, in eval
self.input_ids[self.n_tokens : self.n_tokens + n_tokens] = batch
ValueError: could not broadcast input array from shape (8,) into shape (0,)