More granular selection of which hardware to use for what task
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 30/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- python
- 领域
- ai, backend-api-design
调研方向
该 issue 未指定文件、测试或入口点。首先定位 SDK 中负责模型启动和推理配置的入口点，然后确定是否支持对 Main、Speculative 和 Context 处理进行独立的 CPU/GPU 放置；完成标准应包括所请求的放置控制和 Speculative Decoding 诊断，或一个范围明确的子集。
由索引模型根据 Issue 内容生成。
描述
Would it be possible to add a way to launch/use a model with this configuration?
- Run Main Model inference on CPU (RAM)
- Run Speculative Model on CPU (RAM)
- Run initial context processing on GPU, then move it to CPU (RAM) as well (or only leave this on the GPU if that wouldn’t work)
- Same for the Speculative Model; however, its context size and potential speedup of computation is (probably) negligible.
I am asking because in my testing the speed is fine when running everything on CPU (RAM), including speculative decoding—except for the initial context processing. This step requires a relatively low amount of VRAM but is orders of magnitude faster when computed on the GPU.
Also, as a somewhat unrelated sidenote: I am achieving much higher speeds with these speculative decoding settings (using llama 3.3 70B Q4_K_M with llama 3.2 1B Q8_0, with similar results for other main model sizes):
- Probability: 0.9
- Min: 0
- Max: 5
- Tokens matched: ≥ ~35%
- Speedup: ~1.33x
It would be handy if you could further display all tokens for speculative decoding—both those wasted and those used—along with their respective probabilities, in some kind of debug view to fine-tune the process. Alternatively, an adaptive algorithm could be implemented to determine the best settings either during runtime or as an initial evaluation.
- 主要语言
- Python
- 星标
- 885
- 派生
- 168
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
lmstudio-ai/lmstudio-python 的其他 Issue
-
难度 4/5 3-5 天 新手友好度 25/100
lmstudio-ai/lmstudio-python#186 ·
-
难度 3/5 1-2 天 新手友好度 65/100
lmstudio-ai/lmstudio-python#184 · 1 条评论 · 1 个 reaction ·
-
难度 3/5 1-2 天 新手友好度 45/100
lmstudio-ai/lmstudio-python#183 ·
-
难度 3/5 1-2 天 新手友好度 48/100
lmstudio-ai/lmstudio-python#182 · 1 个 reaction ·
-
error code: 502未关闭
难度 4/5 3-5 天 新手友好度 32/100
lmstudio-ai/lmstudio-python#180 ·
查看 lmstudio-ai/lmstudio-python 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 72/100
slavakurilyak/awesome-ai-agents#845 ·
维护者通常 1 天内回复
-
tests: pin that the atomic writers in utils/paths.py stage the temporary file in the target's directory可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭difficulty:2 enhancement good first issue
难度 1/5 1-3 小时 新手友好度 92/100
MakazhanAlpamys/Soup#2178 ·
维护者通常 1 天内回复
-
[BUG] dcg.waggr permutation p-values are folded twice, so near-zero scores get the smallest p-values未关闭
难度 2/5 1-3 小时 新手友好度 75/100
scverse/rapids-singlecell#854 ·
维护者通常 1 天内回复
-
Slang input: int8_t4_packed and uint8_t4_packed are copied into CrossGL可能已有人在做 @ninglinLiu 今天认领。 未关闭area:frontend backend:slang bug direction:input
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
random_elements(unique=True) on an OrderedDict: ZeroDivisionError once only zero weights remain, AssertionError with use_weighting=False可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 13 天内回复