ggml-org/llama.cpp
server: exit failure if `--embedding` is set with an incoherent `--ubatch-size`
已关闭
#6,263 创建于 2024年3月23日
enhancementgood first issuehelp wanted
仓库指标
- 星标
- (124,339 个星标)
- PR 合并指标
- (平均合并 6天 8小时) (30 天内合并 389 个 PR)
描述
Context
there is no advantage to increase n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.
- See @slaren comment in: https://github.com/ggerganov/llama.cpp/pull/6254#discussion_r1536661327
Proposition
Exit failure if --embedding is set and --ubatch-size != --batch-size in the server example. Probably also in the retrieval example in #6193.
Aldo probably KV bert.context_size must be taken into account.