Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Support Qwen3.8-Flash-Next-GSQ-RCO GGUF and integrate Strata + KVMem

未关闭
  • 哪里不对？
#658 0 条评论 2 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
10/100
Issue 类型
功能
描述清晰度
需要澄清
活跃度
活跃
技术栈
python
领域
machine-learning

调研方向

The payload names no files, tests or entry points. Start by finding the existing model loading and KV cache or memory management code in the repository, and check the linked Roadmap (#79) and FAQ (#84) for any existing plan. The three requests (GGUF support, Strata, KVMem) are separate large integrations, so done would mean a narrowed sub-issue with a stated scope and acceptance test, not a patch to this thread.

由索引模型根据 Issue 内容生成。

描述

Desktop feature
Before you start
  • I have read the Roadmap and this is not already planned there.
  • I have read the FAQ.
  • I have searched existing issues and found no duplicate.
Applies to

Desktop app

What problem does this solve

Hi FreeToken team,

I would like to request support for Qwen3.8-Flash-Next-GSQ-RCO GGUF. Many users are waiting to run this model locally, and FreeToken's edge-native MoE design looks like a great fit. Native GGUF support would make it much easier to load, test, and use this model without extra conversion work.

I also hope you can consider merging/integrating the Strata inference framework. Strata is built for MoE models that cannot fully fit in VRAM, splitting weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and could help run large models like Qwen3.8-Flash-Next on consumer hardware.

Please also consider integrating KVMem. Long-context agent workflows can quickly run out of VRAM, while KVMem keeps old KV history in host RAM and only loads relevant blocks into a bounded GPU working set. That would greatly improve long conversations, tool calls, and long-running sessions in FreeToken.

We really need these features. If any of them are already planned, please let us know. I am happy to help test or provide more details. Thank you!

Proposed solution

Add native support for Qwen3.8-Flash-Next-GSQ-RCO GGUF, so users can load and run this model directly in FreeToken without extra conversion work.

Integrate the Strata inference framework into FreeToken. Strata is designed for MoE models that cannot fully fit in VRAM. It splits weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and would help run large models like Qwen3.8-Flash-Next on consumer hardware.

Integrate KVMem into FreeToken as an optional KV cache / context manager. KVMem keeps long KV history in host RAM and only loads relevant blocks into a bounded GPU working set. This would improve long-context chat, tool calls, and long-running agent sessions without using too much VRAM.

Ideally, expose these features as config options so users can enable or disable them. Please also keep compatibility with FreeToken's current caching and memory management. I am happy to help test if needed.

Alternatives considered

No response

主要语言
Python
星标
14.2k
派生
1.4k
平均合并
21 小时 39 分钟
30 天内合并 PR
100

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

FlashML-org/FreeToken 的其他 Issue

查看 FlashML-org/FreeToken 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。