ggml-org/llama.cpp

server: avoid full prompt eval when 'prompt >= ctx'

Closed

#6,855 opened on Apr 23, 2024

View on GitHubContributor guide
 (3 comments) (0 reactions) (0 assignees)C++ (21,737 forks)batch import
enhancementgood first issue

Repository metrics

Stars
 (124,043 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

When using the server for multi-turn chat, soon or later the prompt is going to surpass the context size, the current approach truncate the prompt by half of the context size excluding n_keep:

https://github.com/ggerganov/llama.cpp/blob/192090bae47960f0d38d4967abe398a5d190057e/examples/server/server.cpp#L1969-L1983

By doing that, common_part is going to match only n_keep tokens (when cache_prompt: true):

https://github.com/ggerganov/llama.cpp/blob/192090bae47960f0d38d4967abe398a5d190057e/examples/server/server.cpp#L2011-L2016

Technically, this is not a full prompt eval, n_keep is not revaluated, but it would be better to avoid this if possible, specially because prompt eval is slow on CPU.

Contributor guide

Research direction
Examine the referenced code in server.cpp lines 1969-1983 and 2011-2016 where prompt truncation occurs when prompt exceeds context size. The goal is to avoid unnecessary full prompt evaluation by adjusting the truncation logic or cache mechanism. Review the issue comments for any maintainer hints or alternative approaches. Consider implementing a check to reuse existing cache when possible without full re evaluation.
Tech stack
cpp
Domain
backend
Issue type
Performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Familiarity with llama.cpp server codeUnderstanding of prompt caching and context size
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30