Embedding document fails if tokens length > n_ctx

The error occurs in llama_cpp/llama.py, specifically in the eval method at line 549. Start by examining the create_embedding method (line 869) and the embed method (line 905) to understand the token handling. The fix likely involves adding a check for token length against n_ctx before calling eval, and raising a clear error or warning. Look for existing validation patterns in the codebase and test with a long document to reproduce the issue.

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [X ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ X] I carefully followed the README.md.
• [ X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [ X] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Embedding a document, if embedding document tokens are > to n_ctx , it should warn or exit()

Current Behavior

it fails with error
File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in embed_documents
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 905, in embed
return list(map(float, self.create_embedding(input)["data"][0]["embedding"]))
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 869, in create_embedding
self.eval(tokens)
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 549, in eval
self.input_ids[self.n_tokens : self.n_tokens + n_tokens] = batch
ValueError: could not broadcast input array from shape (8,) into shape (0,)

Environment and Context

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

document = open("DOC/mybigdocument.txt,"r")
text = document.read()
embeddings = LlamaCppEmbeddings(model_path=model_path,verbose=False)
embeddings.embed_documents([text])

Failure Logs

Pl File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in embed_documents
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/langchain/embeddings/llamacpp.py", line 113, in
embeddings = [self.client.embed(text) for text in texts]
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 905, in embed
return list(map(float, self.create_embedding(input)["data"][0]["embedding"]))
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 869, in create_embedding
self.eval(tokens)
File "/home/mac/chatest/lib/python3.10/site-packages/llama_cpp/llama.py", line 549, in eval
self.input_ids[self.n_tokens : self.n_tokens + n_tokens] = batch
ValueError: could not broadcast input array from shape (8,) into shape (0,)