ggml-org/llama.cpp

llama : refactor the llm.build_xxx functions

Closed

#5,239 opened on Jan 31, 2024

View on GitHubContributor guide
 (3 comments) (3 reactions) (0 assignees)C++ (21,764 forks)batch import
good first issuerefactoringroadmap

Repository metrics

Stars
 (124,134 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Now that we support a large amount of architectures, we can clearly see the patterns when constructing the compute graphs - i.e. optional biases, different norm types, QKV vs Q+K+V, etc.

We should deduplicate the copy-paste portions in functions such as llm.build_llama(), llm.build_falcon(), etc.

The advantage of the current code is that it is easy to look into the graph of a specific architecture. When we refactor this, we will lose this convenience to some extend. So we should think about making this refactoring in such a way that we don't completely obscure which parts of the graph belong to which architectures

Open for ideas and suggestions how to do this best

Contributor guide

Research direction
Examine the existing build functions (e.g., llm.build llama, llm.build falcon) to identify common patterns such as optional biases, norm types, and QKV variations. Propose a refactoring approach that reduces duplication while preserving architecture specific readability. Consider using template metaprogramming or configuration structs.
Tech stack
cpp
Domain
backendmachine learning
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
C++LLM architecture knowledge
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30