ggml-org/llama.cpp
server: exit failure if `--embedding` is set with an incoherent `--ubatch-size`
Fermée
#6 263 ouverte le 23 mars 2024
enhancementgood first issuehelp wanted
Métriques du dépôt
- Stars
- (124 222 étoiles)
- Métriques de merge PR
- (Merge moyen 6j 8h) (389 PRs mergées en 30 j)
Description
Context
there is no advantage to increase n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.
- See @slaren comment in: https://github.com/ggerganov/llama.cpp/pull/6254#discussion_r1536661327
Proposition
Exit failure if --embedding is set and --ubatch-size != --batch-size in the server example. Probably also in the retrieval example in #6193.
Aldo probably KV bert.context_size must be taken into account.