Description
Describe the feature you'd like Certain embedding models produce so called "Matryoshka Embeddings". These embedding vectors have the most information in the first digits of the vector, and the further we go to the back of the vector the less information the digits convey.
This means, that I can take an embedding with e.g. 1024 dimensions, cut off the last 768 dimensions and keep only the first 256 dimensions (or first 100, first 500 etc.). These first 256 have more meaning than the cut off 768 dimensions. This means that instead of storing 1024 dimensions, I can store only 256 dimensions which requires much less space, without loosing a lot of information.
Right now, I cannot use a different dimension when upserting than the embedding model yields via the DocStore. I guess it should be easy to implement. Instead of throwing an error ...
Status: 500
Error: documentStoreServices.insertIntoVectorStoreMiddleware - Error: documentStoreServices.insertIntoVectorStore - Error: documentStoreServices._insertIntoVectorStoreWorkerThread - Error: 400 Bad Request: Wrong input: Vector dimension error: expected dim: 256, got 1024
... let me specify the number of dimensions that I wanna store and cut off the rest:
Additional context Read more about the marvelous matryoshka embeddings on HF: https://huggingface.co/blog/matryoshka