BYOK custom providers: CLI forces greedy sampling (temperature=0) causing reasoning-model degeneration and silent hangs on context overflow

The bug is in the CLI's request generation for BYOK providers. Start by examining the code that adds sampling parameters (temperature, top_p) to chat-completion requests, likely introduced in version 1.0.81. Look for where model IDs are set for utility calls to ensure the configured BYOK model name is used instead of an internal ID like 'gpt-5.4-nano'. Check the compaction logic to understand why it hangs or produces contentless responses after truncation. Reproduce with a custom provider setup and compare request bodies between versions.

Describe the bug

For BYOK / custom OpenAI-compatible providers (e.g. a small thinking model such as qwen-27b served by vLLM), CLI 1.0.81 and later send temperature: 0, top_p: 0.95, frequency_penalty: 0, presence_penalty: 0 in every chat-completion request. CLI 1.0.80 sends no sampling parameters at all, letting the server use its own defaults.

With temperature: 0 (greedy decoding) forced on a thinking model, the model degenerates into long, repetitive, pure-reasoning outputs. Those oversized reasoning outputs (10.6k–25.5k tokens in a single call, verified in session logs) push the request past the provider's hard max_prompt_tokens limit (131,072), triggering compaction + truncation — after which the client either hangs with no error and no further model call (22 minutes until the user killed the process) or ends the turn with a contentless reasoning-only finish_reason: stop response and goes idle with no user-visible signal.

Secondary symptom observed in the same sessions: during compaction / auto-mode utility calls, the CLI sends the internal model id gpt-5.4-nano instead of the configured BYOK model name to the custom endpoint, producing a silent 404.

This is a CLI regression, not a model problem: the same model and the same server run fine on 1.0.80.

Affected version

1.0.81 – 1.0.87 (regression introduced in 1.0.81; verified by wire capture of 1.0.80/1.0.81/1.0.82/1.0.86/1.0.87 — 1.0.80 is healthy)

Steps to reproduce the behavior

1. Configure a custom OpenAI-compatible provider (BYOK):

   COPILOT_PROVIDER_TYPE=openai
   COPILOT_PROVIDER_BASE_URL=<vLLM endpoint>/v1
   COPILOT_PROVIDER_API_KEY=<any>
   COPILOT_PROVIDER_MAX_PROMPT_TOKENS=131072
   COPILOT_PROVIDER_MAX_OUTPUT_TOKENS=8192
   COPILOT_MODEL=qwen-27b
   

2. Run a sustained multi-step task with CLI ≥ 1.0.81 (small tasks like "create fib.py with memoization, run it, verify output" complete fine — degeneration manifests under sustained load, i.e. over many hours / compactions).
3. Observe: repetitive pure-reasoning output; context crossing the 131,072 limit; then either a silent hang (no error, no further model call, for 22+ minutes) or a contentless reasoning-only turn ending the turn with no signal.

Minimal wire-level reproduction of the sampling regression (no long task needed): capture the request bodies of 1.0.80 vs 1.0.81 sent to the same endpoint for the same task. 1.0.80 sends no sampling parameters; 1.0.81 sends temperature: 0, top_p: 0.95, frequency_penalty: 0, presence_penalty: 0.

Expected behavior

• Custom / BYOK providers are not pinned to sampling parameters (1.0.80 behavior: none sent, server defaults apply) — or sampling parameters are configurable via provider settings. temperature: 0 is harmful for thinking models.
• Requests never exceed max_prompt_tokens: the compaction decision accounts for max_output_tokens headroom, so the CLI compacts before overflow rather than truncating after.
• After a truncation/compaction, the CLI issues the next model call or surfaces an error — it never sits silent.
• Contentless reasoning-only turns (large reasoning, 0 content, 0 tool calls, finish_reason: stop) are detected and surfaced or retried with adjusted parameters.
• Utility/compaction calls use the configured BYOK model name, not an internal id (no 404 for gpt-5.4-nano).

Additional context

Wire-level request diff (socket capture, identical task, identical model)

Task: "create fib.py with memoization, run it, verify output" — non-interactive mode, BYOK qwen-27b via an OpenAI-compatible vLLM endpoint.

1.0.80 request body (top-level keys):

model, messages, tools, stream, stream_options

No sampling parameters — server defaults apply.

1.0.87 request body (top-level keys):

model, messages, temperature: 0, top_p: 0.95, frequency_penalty: 0, presence_penalty: 0, tools, stream, stream_options

Version pinning — all five versions captured running the identical task against the identical endpoint:

CLI version	Top-level request keys (sampling params)
1.0.80	model, messages, tools, stream, stream_options — no sampling params
1.0.81	+ temperature: 0, top_p: 0.95, frequency_penalty: 0, presence_penalty: 0 (+ snippy: {enabled: false})
1.0.82	same as 1.0.81 (incl. snippy stub)
1.0.86	same as 1.0.81 (no snippy)
1.0.87	same as 1.0.81 (no snippy)

The regression was introduced in 1.0.81 — the same release that added the interrupted-session restore mechanism (#4673).

Both versions replay assistant reasoning text into the message history (standard for thinking models; not the differentiator).

Session log evidence (2026-09-22, all qwen-27b)

Session	CLI	What happened
26a9ef8c	1.0.80	475 model calls over 22 h, 11 compactions (max 118.5k, never over limit), 0 truncations, 0 hangs. Work completed, PR merged. Max reasoning call 9,754 tokens — productive.
8bdd58ac	1.0.87	User: "you are in a reasoning loop" (02:22)
dd3a60cd	1.0.87	User: "you stuck in a reasoning loop" (04:31); 6 compactions (up to 123.5k — near the 131k limit)
8278f5c4	1.0.87	User: "you are in a reasoning loop" (04:48). Compaction call 404'd once (gpt-5.4-nano does not exist — see below), then an empty 10,662-token pure-reasoning turn (finish_reason: stop, 0 content, 0 tool calls) at 04:58; client went silent for 25 min until user kill.
bf54a1ec	1.0.87	25,510-token degenerate reasoning call (04:56–05:05) → context 131,805 (> 131,072 limit) → compaction_start + truncation 131,805→122,893 at 05:05:21 → no compaction_complete, no further model call → 22-minute silent hang until user kill (05:27).

Causal chain

1. Forced temperature: 0 → greedy decoding → repetition collapse in the thinking model's reasoning trace.
2. Oversized reasoning outputs (and full reasoning replay in history) exceed the provider's hard prompt limit between compaction-threshold checks (threshold ≈ 113k, limit 131k; a single 25k-token call crosses it).
3. Context overflow → truncation. Client then either:
   • (a) hangs: compaction starts, truncation applied, but no compaction-complete and no next model call (bf54a1ec), or
   • (b) emits a contentless reasoning-only finish_reason: stop turn and goes idle with no user-visible signal (8278f5c4).

Secondary bug: utility model calls 404 for BYOK

During compaction (and for auto-mode utility calls) the CLI sends the internal model id gpt-5.4-nano instead of the BYOK model name to the custom endpoint:

model.turn_failed
  model: gpt-5.4-nano   (modelInfo: {id: "gpt-5.4-nano", name: "qwen-27b"})
  error: 404 The model `gpt-5.4-nano` does not exist.

The turn silently recovers (falls back to truncation), but the failure is invisible to the user. Related: #4445 (auto mode picks unavailable models).

Environment

• Operating system: Ubuntu 26.04 LTS, x86_64, bash
• Model server: vLLM 0.28.0 (OpenAI-compatible), max_model_len 262144
• Related issues: #4673 (auto-restore mechanism from the same 1.0.81 release — fixed/disabled by default since 1.0.83), #4445 (auto mode model availability)