ggml-org/llama.cpp
server: exit failure if `--embedding` is set with an incoherent `--ubatch-size`
已關閉
#6,263 建立於 2024年3月23日
enhancementgood first issuehelp wanted
倉庫指標
- 星標
- (124,194 顆星)
- PR 合併指標
- (平均合併 6天 8小時) (30 天內合併 389 個 PR)
描述
Context
there is no advantage to increase n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.
- See @slaren comment in: https://github.com/ggerganov/llama.cpp/pull/6254#discussion_r1536661327
Proposition
Exit failure if --embedding is set and --ubatch-size != --batch-size in the server example. Probably also in the retrieval example in #6193.
Aldo probably KV bert.context_size must be taken into account.