Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

leaderboard: the AI models are ranked under different thinking and temperature settings

Open
  • Is something wrong?
#666 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
python
Domain
data-visualization

Research direction

Read dashboard/config/leaderboard.json, backtest_harness.py, providers/commonstack.py, reasoning_controls.py, and _find_cached_run in domain/leaderboard/service.py to understand how settings are sent and cached. The issue asks maintainers to choose a policy before implementation, so first get that decision; then define done around applying the chosen policy and making the leaderboard communicate the settings used.

Written by the indexing model from the issue text.

Description

The problem

The Competition Leaderboard ranks seven AI models side by side, but they don't all run under the same settings. From dashboard/config/leaderboard.json:

Model Thinking Temperature (randomness)
Claude Haiku 4.5 provider default provider default
Claude Sonnet 4.6 provider default provider default
GPT-5.5 provider default provider default
Gemini 3.1 Pro provider default provider default
DeepSeek V4 Pro off (reasoning_effort: "none") 0
Qwen3.7 Plus off (reasoning_effort: "none") 0
Nemotron 3 Nano off (reasoning_effort: "none") 0

So three models are told not to think and to answer with as little randomness as possible. The other four run however their provider sets them by default.

Checked in the code:

  • The four default models send no temperature. request_trading_decision in backtest_harness.py only adds temperature when one is set.
  • They also send no thinking control. With no reasoning_effort, wants_thinking_off (reasoning_controls.py) means "send nothing", so the provider default applies.
  • The other three go to ChatCompletionsClient (providers/commonstack.py), which always sends "thinking disabled", with temperature 0.
  • Nothing fills in a default in between. OPENROUTER_REASONING_EFFORT only applies to OpenRouter, and all seven entries use CommonStack.

The four "default" models are probably not equal to each other either. From general provider behaviour, not confirmed with a live call: Claude models don't think unless asked to, while GPT-5.5 and Gemini 3.1 Pro usually reason by default. So the board may really be comparing three setups, not two.

Why it matters

If one model may think before answering and another may not, a difference in returns can come from the settings rather than the model. The board presents them as one fair comparison.

How it got this way

Thinking was turned off for DeepSeek and Qwen for reliability, not fairness. The providers/commonstack.py docstring records why. With thinking on, DeepSeek sometimes spent its whole output on thinking and returned no answer. That emptied 13 of 132 leaderboard steps and failed the H6 95% coverage check. The fix was right for those models, but it left the board comparing different setups.

What is shown today
  • A run's results panel has a "Sampling" row: Entry config · …, or Entry config · integration default when nothing is set (app.js:10382).
  • The leaderboard table itself doesn't flag that the models ran under different settings.
  • For the four default models the run stores None for both settings, so there's no record of what the provider actually used.
Things to know before fixing it
  • Changing an entry's settings doesn't re-run it. _find_cached_run (domain/leaderboard/service.py) matches a saved run on strategy, dates, session, capital and strategy prompt, but not on temperature or thinking. After a config change, the board keeps showing the old curve until a forced refresh (POST /api/v1/leaderboard/refresh?force=true).
  • On CommonStack, the leaderboard can only send thinking "off" or the provider default. A graduated value like "low" raises UnsupportedReasoningEffort (reasoning_controls.py). GPT-5.5 therefore can't be pinned to the low effort that paid user backtests use (execution_catalog.py), so leaderboard GPT-5.5 and a user's GPT-5.5 backtest run under different settings.
Decision needed — maintainers, please choose a policy

I'd like to fix this, but which policy to use is your call. Please reply with 1, 2 or 3 (or another approach), and I'll open the PR:

  1. Same settings for everyone. Thinking off (or as low as each model allows), and temperature 0 where the model accepts it. This is the easiest to compare. It may handicap reasoning models, and on CommonStack today "as low as allowed" means fully off or provider default only.
  2. Each model at its own best settings, clearly labelled. This is realistic, but it compares setups rather than models.
  3. Show both. For example, two boards, or a "settings" column and filter.
Dominant language
Python
Stars
767
Forks
156
Avg merge
1d 19h
Merged PRs (30d)
71

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Open-Finance-Lab/AgenticTrading

All issues in Open-Finance-Lab/AgenticTrading

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.