Thinking toggle support for Qwen related models

Open
#2,063 10 comments 3 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python

Research direction

Look at the llama.cpp upstream changes for thinking toggle support in Qwen models. Examine the Python bindings for how model parameters and chat template kwargs are currently handled. Start by finding where Qwen model interactions are defined and how reasoning_budget or similar flags could be integrated. Check existing tests for model parameter handling to understand the pattern.

Written by the indexing model from the issue text.

Description

Is your feature request related to a problem? Please describe.
Cannot toggle thinking in Qwen models, when we do it through the user prompt way, it still gives out opening and closing think tags.

Describe the solution you'd like
Support for params/ flags like reasoning_budget, chat_template_kwargs parameters.
Describe alternatives you've considered
N/A

Additional context
This problem has been a long point of discussion in the upstream project (llama.cpp) too and I am hoping for it to trickle down here and have support for it, noting that the PRs have been merged and llama.cpp has controls over thinking now.

Dominant language
Python
Stars
10.7k
Forks
1.5k
Avg merge
1h 40m
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from abetlen/llama-cpp-python

All issues in abetlen/llama-cpp-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.