Support Qwen3.8-Flash-Next-GSQ-RCO GGUF and integrate Strata + KVMem
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 10/100
- Tipo de issue
- Nueva funcionalidad
- Claridad
- Necesita aclaración
- Estado de actividad
- Activo
- Stack tecnológico
- python
- Área
- machine-learning
Línea de trabajo
The payload names no files, tests or entry points. Start by finding the existing model loading and KV cache or memory management code in the repository, and check the linked Roadmap (#79) and FAQ (#84) for any existing plan. The three requests (GGUF support, Strata, KVMem) are separate large integrations, so done would mean a narrowed sub-issue with a stated scope and acceptance test, not a patch to this thread.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Before you start
- I have read the Roadmap and this is not already planned there.
- I have read the FAQ.
- I have searched existing issues and found no duplicate.
Applies to
Desktop app
What problem does this solve
Hi FreeToken team,
I would like to request support for Qwen3.8-Flash-Next-GSQ-RCO GGUF. Many users are waiting to run this model locally, and FreeToken's edge-native MoE design looks like a great fit. Native GGUF support would make it much easier to load, test, and use this model without extra conversion work.
I also hope you can consider merging/integrating the Strata inference framework. Strata is built for MoE models that cannot fully fit in VRAM, splitting weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and could help run large models like Qwen3.8-Flash-Next on consumer hardware.
Please also consider integrating KVMem. Long-context agent workflows can quickly run out of VRAM, while KVMem keeps old KV history in host RAM and only loads relevant blocks into a bounded GPU working set. That would greatly improve long conversations, tool calls, and long-running sessions in FreeToken.
We really need these features. If any of them are already planned, please let us know. I am happy to help test or provide more details. Thank you!
Proposed solution
Add native support for Qwen3.8-Flash-Next-GSQ-RCO GGUF, so users can load and run this model directly in FreeToken without extra conversion work.
Integrate the Strata inference framework into FreeToken. Strata is designed for MoE models that cannot fully fit in VRAM. It splits weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and would help run large models like Qwen3.8-Flash-Next on consumer hardware.
Integrate KVMem into FreeToken as an optional KV cache / context manager. KVMem keeps long KV history in host RAM and only loads relevant blocks into a bounded GPU working set. This would improve long-context chat, tool calls, and long-running agent sessions without using too much VRAM.
Ideally, expose these features as config options so users can enable or disable them. Please also keep compatibility with FreeToken's current caching and memory management. I am happy to help test if needed.
Alternatives considered
No response
- Lenguaje dominante
- Python
- Estrellas
- 14.2k
- Forks
- 1.4k
- Merge medio
- 21 h 39 min
- PR fusionados (30 d)
- 100
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de FlashML-org/FreeToken
-
Desktop feature
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
FlashML-org/FreeToken#800 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
FlashML-org/FreeToken#767 ·
Los mantenedores suelen responder en 1 día
-
test_gemv_path_matches_the_transcription fails on sm_120 (bf16 rounding tie exceeds the tolerance)Posiblemente ocupada @void-mckenzie la tomó hace 17 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
FlashML-org/FreeToken#649 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
FlashML-org/FreeToken#648 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
FlashML-org/FreeToken#646 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de FlashML-org/FreeToken
Issues similares
-
infrastructure
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
NatLabRockies/H2Integrate#977 ·
Los mantenedores suelen responder en 1 día
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
topoteretes/cognee#6470 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
-
area:docs priority:p2 type:docs wave:9
Dificultad 1/5 Menos de una hora Aptitud para principiantes 84/100
jejjohnson/radtrax#293 ·
Los mantenedores suelen responder en 1 día
-
bug core plugins
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
bunkerity/bunkerweb#4121 · 1 comentario ·
Los mantenedores suelen responder en 1 día