Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Running local LLMs

未关闭 适合新手
  • 哪里不对？
#176 1 条评论 3 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
64/100
Issue 类型
文档
描述清晰度
基本清楚
活跃度
活跃
技术栈
ollama
领域
ai, documentation

调研方向

从现有的 issue 文本及其链接的架构图开始，然后查看对 Ollama、vLLM 和 SGLang 的描述。根据注释中的要求，在概览中补充更深入的洞察，同时让比较内容对正在选择本地 LLM 引擎的人来说易于理解。当文档中记录的差异和技术在当前大致总结的基础上得到扩展时，即表示完成。

由索引模型根据 Issue 内容生成。

描述

Ai discuss enhancement

Brief overview of Ollama, vLLM and SGLang

To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang.
Each engine handles requests differently. The diagram below shows the differences and the main techniques behind each engine.

Image

  • Ollama: Ollama is best for local dev, prototyping, and laptop-scale hardware. The architecture is inherently sequential. A local user calls the OpenAI-compatible API, and requests line up in a FIFO queue. Then Ollama runs a pre-quantized GGUF model, a compressed format it pulls, and the response comes back to the user.

  • vLLM: vLLM is best for high-traffic serving, max GPU utilization, and thousands of concurrent requests. Many users hit the server at once, and continuous batching slots new requests into the running batch instead of making them wait for it to finish. PagedAttention stores the KV cache, the memory a model keeps for tokens it has already processed. The PagedAttention maps the OS memory pages to vLLM memory blocks.

  • SGLang: SGLang is best for AI agents and tool loops, multi-turn chats, and JSON/regex outputs. The most commun example is when the workflow involves using repeated context, like a static system prompt or a large RAG documents, during a CI workflow. Agents and multi-turn chats send requests whose prompts overlap heavily. A prefix-aware scheduler routes them through the RadixAttention cache, a radix tree that reuses every shared prefix instead of recomputing it.

[!NOTE]
This is for the big picture. Needs to be continued with a bit more in deep insights...

主要语言
JavaScript
星标
291
派生
25
30 天内没有已合并 PR

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

dwyl/technology-stack 的其他 Issue

查看 dwyl/technology-stack 的全部 Issue

相似的 Issue

更多 JavaScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。