ggml-org/llama.cpp
server: exit failure if `--embedding` is set with an incoherent `--ubatch-size`
Chiusa
#6263 aperta il 23 mar 2024
enhancementgood first issuehelp wanted
Metriche repository
- Star
- (124.222 stelle)
- Metriche merge PR
- (Merge medio 6g 8h) (389 PR mergiate in 30 g)
Descrizione
Context
there is no advantage to increase n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.
- See @slaren comment in: https://github.com/ggerganov/llama.cpp/pull/6254#discussion_r1536661327
Proposition
Exit failure if --embedding is set and --ubatch-size != --batch-size in the server example. Probably also in the retrieval example in #6193.
Aldo probably KV bert.context_size must be taken into account.