simonw/llm

Embeddings need to handle truncation automatically (maybe via extra options)

Open

#1,085 opened on May 25, 2025

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (877 forks)github user discovery
embeddingsenhancementhelp wanted

Repository metrics

Stars
 (12,006 stars)
PR merge metrics
 (PR metrics pending)

Description

I frustrating thing about embeddings right now is this:

cd llm
llm embed-multi --files docs/ '*.md' -d /tmp/docs.db docs

This fails with a messy error:

...
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/openai/resources/embeddings.py", line 128, in create
    return self._post(
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/openai/_base_client.py", line 1242, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/openai/_base_client.py", line 919, in request
    return self._request(
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/openai/_base_client.py", line 1023, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 17143 tokens (17143 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}

That error reformatted:

{
  "error": {
    "message": "This model's maximum context length is 8192 tokens, however you requested 17143 tokens (17143 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

NO embeddings are stored in the /tmp/docs.db database. The entire operation just fails, with no obvious way to recover.

Contributor guide