ggml-org/llama.cpp
server: exit failure if `--embedding` is set with an incoherent `--ubatch-size`
クローズ
#6,263 opened on 2024/03/23
enhancementgood first issuehelp wanted
Repository metrics
- Stars
- (124,222 個のスター)
- PR merge metrics
- (平均マージ 6d 8h) (30d で 389 merged PRs)
説明
Context
there is no advantage to increase n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.
- See @slaren comment in: https://github.com/ggerganov/llama.cpp/pull/6254#discussion_r1536661327
Proposition
Exit failure if --embedding is set and --ubatch-size != --batch-size in the server example. Probably also in the retrieval example in #6193.
Aldo probably KV bert.context_size must be taken into account.