Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Support Qwen3.8-Flash-Next-GSQ-RCO GGUF and integrate Strata + KVMem

Aperta
  • Qualcosa non va?
#658 0 commenti 2 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
10/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning

Direzione di ricerca

The payload names no files, tests or entry points. Start by finding the existing model loading and KV cache or memory management code in the repository, and check the linked Roadmap (#79) and FAQ (#84) for any existing plan. The three requests (GGUF support, Strata, KVMem) are separate large integrations, so done would mean a narrowed sub-issue with a stated scope and acceptance test, not a patch to this thread.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Desktop feature
Before you start
  • I have read the Roadmap and this is not already planned there.
  • I have read the FAQ.
  • I have searched existing issues and found no duplicate.
Applies to

Desktop app

What problem does this solve

Hi FreeToken team,

I would like to request support for Qwen3.8-Flash-Next-GSQ-RCO GGUF. Many users are waiting to run this model locally, and FreeToken's edge-native MoE design looks like a great fit. Native GGUF support would make it much easier to load, test, and use this model without extra conversion work.

I also hope you can consider merging/integrating the Strata inference framework. Strata is built for MoE models that cannot fully fit in VRAM, splitting weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and could help run large models like Qwen3.8-Flash-Next on consumer hardware.

Please also consider integrating KVMem. Long-context agent workflows can quickly run out of VRAM, while KVMem keeps old KV history in host RAM and only loads relevant blocks into a bounded GPU working set. That would greatly improve long conversations, tool calls, and long-running sessions in FreeToken.

We really need these features. If any of them are already planned, please let us know. I am happy to help test or provide more details. Thank you!

Proposed solution

Add native support for Qwen3.8-Flash-Next-GSQ-RCO GGUF, so users can load and run this model directly in FreeToken without extra conversion work.

Integrate the Strata inference framework into FreeToken. Strata is designed for MoE models that cannot fully fit in VRAM. It splits weights across GPU VRAM, host RAM, and NVMe while keeping at least 4-bit precision. This fits FreeToken's goal of efficient local inference and would help run large models like Qwen3.8-Flash-Next on consumer hardware.

Integrate KVMem into FreeToken as an optional KV cache / context manager. KVMem keeps long KV history in host RAM and only loads relevant blocks into a bounded GPU working set. This would improve long-context chat, tool calls, and long-running agent sessions without using too much VRAM.

Ideally, expose these features as config options so users can enable or disable them. Please also keep compatibility with FreeToken's current caching and memory management. I am happy to help test if needed.

Alternatives considered

No response

Lingua principale
Python
Stelle
14.2k
Fork
1.4k
Merge medio
21h 17m
PR unite (30g)
97

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FlashML-org/FreeToken

Tutte le issue di FlashML-org/FreeToken

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.