Benchmark uses unsupported temperature parameter with reasoning models

Open
#3 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
50/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
machine-learning

Research direction

Locate the benchmark entry point where requests are built and temperature is passed to every model. Check how the default gpt-5 reasoning model is identified, then run the benchmark or its existing evaluation checks to confirm reasoning models no longer receive temperature or similar sampling parameters.

Written by the indexing model from the issue text.

Description

Description

The benchmark currently passes the temperature parameter to all models by default.

However, reasoning models such as gpt-5 do not support custom temperature values and only allow the default value. As a result, evaluation fails with errors like:

Error code: 400 - Unsupported value: 'temperature' does not support 0.0 with this model.

gpt-5 is used as model by default.

Expected behavior

Do not send temperature (and similar sampling parameters) to reasoning models.

Dominant language
Python
Stars
28
Forks
0
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from VisionXLab/Rise-Video

All issues in VisionXLab/Rise-Video

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.