llama : combined beam search + grammar sampling strategy
#2.923 geöffnet am 31.08.2023
Repository-Metriken
- Stars
- (124.134 Sterne)
- PR-Merge-Metriken
- (Durchschn. Merge 6T 8h) (389 gemergte PRs in 30 T)
Beschreibung
This feature was proposed by @spion in https://github.com/ggerganov/llama.cpp/issues/2813#issuecomment-1694390583
In some cases, its useful to do constrained evaluation of logits based on a union of possible text values, then pick the sum { logits } (i.e. product(probabilities)) that gives the most probable outcome overall.
E.g. template (using MS guidance)
{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}
To definitely make the best choice, we'd need to calculate the probability of all 3 token sequences. Its easy if all the choices map to a single token, but with multiple tokens we'd need not just parallel generation but parallel logit evaluation of multiple possible paths.
If we go greedy, we might get suboptimal results in cases multiple choices start with the same logit.
It should be possible to implement this by combining the existing beam search and grammar sampling features. See the discussion in the referenced comment for more info