[Bug]: Runtime text-generation fallback drops the configured custom model and invokes Haiku

Start in apps/server/src/textGeneration/TextGeneration.ts and trace the runtime fallback path that calls fallbackTextGenerationModelSelection; compare its inputs with the settings resolution and the direct helper control in the reproduction. Run the supplied repro.sh to confirm the selected model and recorded Claude CLI arguments. Done when runtime fallback preserves the configured custom model and the reproduction’s controls still pass.

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

Use a checkout of the commit under Version or commit with its lockfile dependencies installed. Save the script below outside the checkout as repro.sh and run env -u NODE_ENV sh /path/to/repro.sh /path/to/checkout on macOS or Linux. It creates a private HOME, TMPDIR, T3CODE_HOME and Claude home under a mktemp directory (set REPRO_TMP_ROOT to choose its parent), installs one temporary test at apps/server/src/textGeneration/verifyCustomModelFallback.test.ts (refusing to overwrite an existing file), runs it with vp test run, and removes the test and fixture on exit. No live provider credentials are needed.

The test calls the exported production TextGeneration.make with the settings-resolved input that ThreadTitleRegenerationService passes, the real ProviderInstallationGate layer and the production Claude text-generation driver. Provider snapshots, the instance registry and the Codex text-generation result are fixtures. The Claude binary is a local Node script that records the actual --model argument and either accepts every model or, in strict mode, rejects any model other than the custom one.

1. Configure two enabled instances: codex and claudeAgent, with customModels: ["z-ai/glm-5.3-flash"] on claudeAgent. Set the thread default (defaultModelSelection) to claudeAgent / z-ai/glm-5.3-flash with thinking: false. Leave the text-generation model unset, so it resolves to the schema default codex / gpt-6-luna with low reasoning effort.
2. Make the Claude instance installed, authenticated and ready. Make the Codex primary exhausted by passing {primary: {usedPercent: 100, windowDurationMins: 300}} through the production codexRateLimitsToLimits converter. Separately, make it not installed, or signed out, through its snapshot.
3. Generate a thread title. In each of those three primary states the fallback dispatches claudeAgent / claude-haiku-5-5 without options.
4. Repeat the exhausted case with the strict CLI fixture that accepts only the custom model. Generation fails with TextGenerationError.
5. Controls in the same run: a ready primary keeps codex / gpt-6-luna; calling the shared helper directly with {defaultModelSelection: customSelection} returns the custom model with thinking: false; settings-time resolution with Codex disabled returns the custom model; explicitly setting the text-generation model to the custom selection succeeds against the strict fixture; a generic primary error status or an invocation error does not trigger fallback; a missing, signed-out or absent fallback instance is skipped, leaving the failing primary selected.

The script exits 0 only when the wrong fallback model, the recorded CLI model arguments, the strict-backend failure and every control match these assertions.

#!/bin/sh
set -eu
checkout=$(cd "${1:-.}" && pwd)
tmp=$(mktemp -d "${REPRO_TMP_ROOT:-${TMPDIR:-/tmp}}/runtime-fallback.XXXXXX")
fixture="$checkout/apps/server/src/textGeneration/verifyCustomModelFallback.test.ts"
if [ -e "$fixture" ]; then echo 'Fixture already exists; refusing to overwrite.' >&2; exit 2; fi
cleanup() { rm -f "$fixture"; rm -rf "$tmp"; }
trap cleanup EXIT INT TERM
mkdir -p "$tmp/userhome" "$tmp/tmp" "$tmp/t3" "$tmp/claude"
cat > "$tmp/claude-fixture.cjs" <<'JS'
const fs = require('node:fs');
const args = process.argv.slice(2);
const model = args[args.indexOf('--model') + 1];
fs.appendFileSync(process.env.VERIFY_ARGV_LOG, JSON.stringify({model})+'\n');
process.stdin.resume();
process.stdin.on('end', () => {
  if (process.env.VERIFY_STRICT === '1' && model !== 'z-ai/glm-5.3-flash') {
    process.stderr.write('Backend rejects unsupported model: '+model);
    process.exitCode = 1;
  } else {
    process.stdout.write(JSON.stringify({structured_output:{title:'Generated with '+model, subject:'Update fixture', body:'Fixture body'}}));
  }
});
JS
cat > "$tmp/claude".sh <<'JS'
#!/bin/sh
exec "$VERIFY_NODE" "$VERIFY_CLI_SOURCE" "$@"
JS
chmod +x "$tmp/claude.sh"
cat > "$fixture" <<'TS'
import { appendFileSync, readFileSync } from "node:fs";
import * as NodeServices from "@effect/platform-node/NodeServices";
import { it } from "@effect/vitest";
import { expect } from "vite-plus/test";
import { ClaudeSettings, ProjectId, ProviderDriverKind, ProviderInstanceId, TextGenerationError, type ModelSelection, type ServerProvider } from "@t3tools/contracts";
import type { ProviderInstance } from "@t3tools/provider-core/server/driver";
import * as Effect from "effect/Effect";
import * as Layer from "effect/Layer";
import * as Schema from "effect/Schema";
import { fallbackTextGenerationModelSelection } from "@t3tools/shared/serverSettings";
import * as ProviderInstallationGate from "../provider/ProviderInstallationGate.ts";
import * as ProviderInstanceRegistry from "../provider/ProviderInstanceRegistry.ts";
import * as ServerSettings from "../serverSettings.ts";
import * as SourceControlProviderRegistry from "../sourceControl/SourceControlProviderRegistry.ts";
import * as ServerConfig from "../config.ts";
import * as TextGeneration from "./TextGeneration.ts";
import * as TextGenerators from "./TextGenerators.ts";
import { makeClaudeTextGeneration } from "./ClaudeTextGeneration.ts";
import { codexRateLimitsToLimits } from "../provider/codexUsageLimits.ts";

const primary = ProviderInstanceId.make("codex");
const fallback = ProviderInstanceId.make("claudeAgent");
const projectId = ProjectId.make("project-a");
const custom: ModelSelection = {instanceId:fallback, model:"z-ai/glm-5.3-flash", options:[{id:"thinking",value:false}]};
const selected: ModelSelection = {instanceId:primary, model:"gpt-6-luna", options:[{id:"reasoningEffort",value:"low"}]};
const instancesConfig = {
  [primary]: {driver:ProviderDriverKind.make("codex"), enabled:true, config:{}},
  [fallback]: {driver:ProviderDriverKind.make("claudeAgent"), enabled:true, config:{customModels:[custom.model]}},
};
const emit = (label: string, value: unknown) => appendFileSync(process.env.VERIFY_RESULT_LOG!,label+" "+JSON.stringify(value)+"\n");
const ready = (instanceId: ProviderInstanceId): ServerProvider => ({instanceId, driver:ProviderDriverKind.make(instanceId === primary ? "codex" : "claudeAgent"), enabled:true, installed:true, status:"ready", auth:{status:"authenticated"}, version:null, checkedAt:"1970-01-01T00:00:00Z", models:[], slashCommands:[], skills:[]});
type State = "ready"|"exhausted"|"not-installed"|"signed-out"|"generic-error"|"invoke-error";
function probe(state: State, strict = false, fallbackState = "ready", explicitCustom = false) {
  return Effect.gen(function*() {
    const calls: ModelSelection[] = [];
    const env = {...process.env, VERIFY_STRICT: strict ? "1":"0"};
    const claude = yield* makeClaudeTextGeneration(Schema.decodeSync(ClaudeSettings)({binaryPath:process.env.VERIFY_CLI, homePath:process.env.VERIFY_CLAUDE_HOME,customModels:[custom.model]}),env);
    const entries: ProviderInstance[] = [primary,fallback].flatMap(instanceId => {
      if (instanceId === fallback && fallbackState === "absent") return [];
      const snap = ready(instanceId);
      if (instanceId === primary) {
        if (state === "exhausted") snap.usageLimits = codexRateLimitsToLimits({checkedAt:"1970-01-01T00:00:00Z",snapshot:{primary:{usedPercent:100,windowDurationMins:300}}});
        if (state === "not-installed") Object.assign(snap,{installed:false,status:"error",auth:{status:"unknown"}});
        if (state === "signed-out") Object.assign(snap,{status:"error",auth:{status:"unauthenticated"}});
        if (state === "generic-error") Object.assign(snap,{status:"error"});
      } else {
        if (fallbackState === "not-installed") Object.assign(snap,{installed:false,status:"error"});
        if (fallbackState === "signed-out") Object.assign(snap,{auth:{status:"unauthenticated"},status:"error"});
      }
      const primaryService = TextGeneration.TextGeneration.of({
        generateCommitMessage: () => Effect.die("unused"), generatePrContent: () => Effect.die("unused"),generateBranchName: () => Effect.die("unused"),generateReviewGuide: () => Effect.die("unused"), generateTranscriptionPostProcessing: () => Effect.die("unused"),
        generateThreadTitle: input => Effect.sync(()=>{calls.push(input.modelSelection);}).pipe(Effect.flatMap(()=>state === "ready" || state === "exhausted" ? Effect.succeed({title:"Primary title"}) : Effect.fail(new TextGenerationError({operation:"generateThreadTitle",detail:"Primary provider cannot run"})))),
      });
      const textGeneration = instanceId === primary ? primaryService : {...claude, generateThreadTitle:(input: Parameters<typeof claude.generateThreadTitle>[0])=>Effect.sync(()=>{calls.push(input.modelSelection);}).pipe(Effect.andThen(claude.generateThreadTitle(input)))};
      return [{instanceId, driverKind:snap.driver,continuationIdentity:{driverKind:snap.driver,continuationKey:instanceId},displayName:undefined,enabled:true,snapshot:{getSnapshot:Effect.succeed(snap)} as ProviderInstance["snapshot"],orchestrationAdapter:{} as ProviderInstance["orchestrationAdapter"],textGeneration}];
    });
    const settingsLayer = ServerSettings.layerTest({providerInstances:instancesConfig,defaultModelSelection:custom,...(explicitCustom ? {textGenerationModelSelection:custom} : {})});
    const service = yield* TextGeneration.make.pipe(Effect.provide(Layer.mergeAll(
      settingsLayer,
      Layer.mock(ProviderInstanceRegistry.ProviderInstanceRegistry)({listInstances:Effect.succeed(entries),getInstance:id=>Effect.succeed(entries.find(entry=>entry.instanceId===id))}),
      Layer.mock(TextGenerators.TextGenerators)({runner:()=>Effect.undefined}),
      Layer.mock(SourceControlProviderRegistry.SourceControlProviderRegistry)({resolveLink:()=>Effect.die("unused")}),
      ProviderInstallationGate.layer,
    )));
    // ThreadTitleRegenerationService passes the settings-resolved text model.
    const settings = yield* ServerSettings.ServerSettingsService.pipe(Effect.flatMap(service=>service.getSettings),Effect.provide(settingsLayer));
    const result = yield* service.generateThreadTitle({cwd:process.env.VERIFY_WORK!,message:"Explain a code change",linkedContext:"",modelSelection:settings.textGenerationModelSelection,projectId}).pipe(Effect.match({onFailure:error=>({error:error.detail}),onSuccess:value=>value}));
    return {calls,result};
  });
}
it.effect("production runtime fallback discards custom model; controls preserve it",()=>Effect.gen(function*(){
  const without = fallbackTextGenerationModelSelection(instancesConfig,id=>id===fallback);
  const withOption = fallbackTextGenerationModelSelection(instancesConfig,id=>id===fallback,{defaultModelSelection:custom});
  emit("DIRECT_WITHOUT_OPTION",without); emit("DIRECT_WITH_OPTION",withOption);
  expect(without).toEqual({instanceId:fallback,model:"claude-haiku-5-5"});expect(withOption).toEqual(custom);
  const settings = yield* ServerSettings.ServerSettingsService.pipe(Effect.flatMap(service=>service.getSettings),Effect.provide(ServerSettings.layerTest({providerInstances:{...instancesConfig,[primary]:{...instancesConfig[primary],enabled:false}},defaultModelSelection:custom})));
  emit("SETTINGS_CONTROL",settings.textGenerationModelSelection);expect(settings.textGenerationModelSelection).toEqual(custom);
  for (const state of ["ready","exhausted","not-installed","signed-out","generic-error","invoke-error"] as const) {
    const value = yield* probe(state);emit("RUNTIME_"+state.toUpperCase(),value);
    const switches = ["exhausted","not-installed","signed-out"].includes(state);
    expect(value.calls).toEqual([switches ? without : selected]);
    if (switches) expect(value.result).toEqual({title:"Generated with claude-haiku-5-5"});
    if (state === "ready") expect(value.result).toEqual({title:"Primary title"});
    if (state === "invoke-error" || state === "generic-error") expect(value.result).toEqual({error:"Primary provider cannot run"});
  }
  const strict = yield* probe("exhausted",true);emit("RUNTIME_CUSTOM_ONLY_BACKEND",strict);expect(strict.result).toEqual({error:"Claude CLI command failed: Backend rejects unsupported model: claude-haiku-5-5"});
  const explicit = yield* probe("exhausted",true,"ready",true);emit("EXPLICIT_CUSTOM_CONTROL",explicit);expect(explicit.calls).toEqual([custom]);expect(explicit.result).toEqual({title:"Generated with z-ai/glm-5.3-flash"});
  for (const fallbackState of ["not-installed","signed-out","absent"]) {
    const value = yield* probe("signed-out",true,fallbackState);emit("FALLBACK_"+fallbackState.toUpperCase(),value);expect(value.calls).toEqual([selected]);expect(value.result).toEqual({error:"Primary provider cannot run"});
  }
  const argv = readFileSync(process.env.VERIFY_ARGV_LOG!,"utf8").trim().split("\n").map(line=>JSON.parse(line));emit("ACTUAL_CLI_MODELS",argv);
  expect(argv).toEqual([{model:"claude-haiku-5-5"},{model:"claude-haiku-5-5"},{model:"claude-haiku-5-5"},{model:"claude-haiku-5-5"},{model:"z-ai/glm-5.3-flash"}]);
  appendFileSync(process.env.VERIFY_RESULT_LOG!,"DEFECT PRESENT: production runtime dispatched Haiku; custom-only backend failed; controls passed.\n");
}).pipe(Effect.provide(ServerConfig.ServerConfig.layerTest(process.env.VERIFY_WORK!,{prefix:"verify-fallback-"}).pipe(Layer.provideMerge(NodeServices.layer)))));
TS
cd "$checkout"
VERIFY_RESULT_LOG="$tmp/results.jsonl" VERIFY_NODE=$(command -v node) VERIFY_CLI_SOURCE="$tmp/claude-fixture.cjs" VERIFY_CLI="$tmp/claude.sh" VERIFY_CLAUDE_HOME="$tmp/claude" VERIFY_ARGV_LOG="$tmp/argv.jsonl" VERIFY_WORK="$tmp" HOME="$tmp/userhome" TMPDIR="$tmp/tmp" T3CODE_HOME="$tmp/t3" PATH="$checkout/node_modules/.bin:$PATH" env -u NODE_ENV vp test run apps/server/src/textGeneration/verifyCustomModelFallback.test.ts
cat "$tmp/results.jsonl"

Expected behavior

When runtime fallback chooses an instance and the configured thread default is one of that instance's custom models, the fallback should use that thread default with its options, as the shared helper does for its settings-time callers. Built-in thread defaults keep the cheap default model. When #19386 added this exception, the helper's doc comment explained it (doc comment in the commit that #19386 merged, 3062701c24):

The model is the driver's cheap default, unless defaultModelSelection (the thread default) is one of that instance's custom models. A custom model usually means a backend that may refuse the built-in default, and the thread default, with its options, is known to work there.

That paragraph has since been shortened away from the current doc comment, but the helper's implementation still keeps the exception when the option is supplied, and the settings regression test pins it with the same z-ai/glm-5.3-flash plus thinking: false selection. Machine settings and project settings pass the option. In the reproduction, the exhausted, not-installed and signed-out cases should dispatch z-ai/glm-5.3-flash with thinking: false, and the strict fixture should return a title.

Actual behavior

Summary. When the primary text-generation provider is exhausted, missing or signed out, the runtime fallback in TextGeneration.make picks another instance but always uses that driver's cheap default model, even when the configured thread default is one of that fallback instance's custom models. A Claude instance whose thread default is the custom model z-ai/glm-5.3-flash is asked for claude-haiku-5-5 instead, and the custom model's options are dropped. A backend that only serves custom models then rejects the request and generation fails; title generation was the exercised operation. The shared helper fallbackTextGenerationModelSelection already supports this case: #19386 added the custom-model exception for its two settings-time callers, before any runtime fallback existed. The runtime fallback was added later by #26164, which reused the helper without passing defaultModelSelection, so the exception never reached that caller. This is a gap introduced when #26164 reused the helper, not a defect in #19386's scope.

The runtime fallback call passes only the instances and an eligibility predicate, so the helper's custom-model branch cannot run and it returns {instanceId: "claudeAgent", model: "claude-haiku-5-5"}. The recorded CLI arguments show --model claude-haiku-5-5 in all four fallback runs. With the permissive fixture the title is Generated with claude-haiku-5-5; with the strict fixture generation fails with Claude CLI command failed: Backend rejects unsupported model: claude-haiku-5-5. Explicitly selecting the custom model for text generation returns Generated with z-ai/glm-5.3-flash against the same strict fixture. A rerun of the script above at the tested commit passed all its assertions with these results.

Fallback is a preflight decision: it runs when the selected provider's usage window is exhausted, or when its probe is in error and it is not installed or explicitly unauthenticated. An arbitrary invocation error, or an error status on an installed and authenticated provider, does not retry another provider. Missing, signed-out or absent fallback instances are skipped, and when none qualifies the original selection stays.

For title generation, the production caller logs Thread title generation failed and completes without a new title. Commit messages, PR text, branch names, review guides and transcript post-processing go through the same runtime wrapper, including the commit-message caller, but only title generation was executed. No live backend, GUI or paid provider request was run, so this report makes no claim about how often real custom-model backends reject the built-in default or about monetary charges.

Evidence

• Expected source: the custom-model exception in packages/shared/src/serverSettings.ts lines 73-100 and its original doc comment from #19386, the machine and project settings callers that pass defaultModelSelection, the regression test in apps/server/src/serverSettings.test.ts lines 803-844, and the helper-with-option and settings controls in the reproduction.
• Failure source: the observed reproduction output under Logs or stack traces, and the call without defaultModelSelection in apps/server/src/textGeneration/TextGeneration.ts lines 128-142.
• Evidence provenance: observed
• Local verification: reproduced
• Reproduction completeness: complete

The run uses production TextGeneration.make, the real installation gate, the production Claude text-generation driver and a spawned local CLI fixture; provider snapshots, the instance registry and Codex execution are fixtures.

Related

Related, not duplicates: #12651 (the settings-time failure that #19386 fixed); #27792 (credits incorrectly triggering fallback); open #31032 (on-demand quota refresh), #34725 (thread-owned title selection) and #33202 (thread model for commit and PR writing), whose patches leave this runtime helper call without defaultModelSelection. #29607 later widened the runtime fallback to missing and signed-out primaries.

Restoration check

With the script's input, the exhausted, not-installed and signed-out cases dispatch claudeAgent / z-ai/glm-5.3-flash with thinking: false, the recorded CLI model is z-ai/glm-5.3-flash, and the strict fixture returns Generated with z-ai/glm-5.3-flash. As controls, a built-in thread default still falls back to the driver's cheap default, a ready primary keeps its selection, a thread default on a different instance than the chosen fallback is not used, and missing, signed-out or absent fallback instances are still skipped. The script asserts the current failing results, so its RUNTIME_*, RUNTIME_CUSTOM_ONLY_BACKEND and ACTUAL_CLI_MODELS expectations need updating to the restored values for this check.

Impact

Minor bug or occasional failure

Version or commit

0.1.9 at d9805b2ca667be7caec8c9035950564ab96704c3, rerun from a clean checkout; an earlier copy of the same script, differing only in the name of its private home directory, also passed in the same checkout switched to upstream main at 6807c3fb865d3f9fcc0679b2e283cfd6064a4ea7, where the relevant runtime and settings files are unchanged

Environment

Darwin arm64, Node v26.12.1, repository-pinned Vite+ 1.4.0 and Vitest 5.0.5; production text-generation service and Claude driver with a local CLI fixture, with no server, browser, desktop app or live provider

Logs or stack traces

The script's output after the test summary:

DIRECT_WITHOUT_OPTION {"instanceId":"claudeAgent","model":"claude-haiku-5-5"}
DIRECT_WITH_OPTION {"instanceId":"claudeAgent","model":"z-ai/glm-5.3-flash","options":[{"id":"thinking","value":false}]}
SETTINGS_CONTROL {"instanceId":"claudeAgent","model":"z-ai/glm-5.3-flash","options":[{"id":"thinking","value":false}]}
RUNTIME_READY {"calls":[{"instanceId":"codex","model":"gpt-6-luna","options":[{"id":"reasoningEffort","value":"low"}]}],"result":{"title":"Primary title"}}
RUNTIME_EXHAUSTED {"calls":[{"instanceId":"claudeAgent","model":"claude-haiku-5-5"}],"result":{"title":"Generated with claude-haiku-5-5"}}
RUNTIME_NOT-INSTALLED {"calls":[{"instanceId":"claudeAgent","model":"claude-haiku-5-5"}],"result":{"title":"Generated with claude-haiku-5-5"}}
RUNTIME_SIGNED-OUT {"calls":[{"instanceId":"claudeAgent","model":"claude-haiku-5-5"}],"result":{"title":"Generated with claude-haiku-5-5"}}
RUNTIME_GENERIC-ERROR {"calls":[{"instanceId":"codex","model":"gpt-6-luna","options":[{"id":"reasoningEffort","value":"low"}]}],"result":{"error":"Primary provider cannot run"}}
RUNTIME_INVOKE-ERROR {"calls":[{"instanceId":"codex","model":"gpt-6-luna","options":[{"id":"reasoningEffort","value":"low"}]}],"result":{"error":"Primary provider cannot run"}}
RUNTIME_CUSTOM_ONLY_BACKEND {"calls":[{"instanceId":"claudeAgent","model":"claude-haiku-5-5"}],"result":{"error":"Claude CLI command failed: Backend rejects unsupported model: claude-haiku-5-5"}}
EXPLICIT_CUSTOM_CONTROL {"calls":[{"instanceId":"claudeAgent","model":"z-ai/glm-5.3-flash","options":[{"id":"thinking","value":false}]}],"result":{"title":"Generated with z-ai/glm-5.3-flash"}}
FALLBACK_NOT-INSTALLED {"calls":[{"instanceId":"codex","model":"gpt-6-luna","options":[{"id":"reasoningEffort","value":"low"}]}],"result":{"error":"Primary provider cannot run"}}
FALLBACK_SIGNED-OUT {"calls":[{"instanceId":"codex","model":"gpt-6-luna","options":[{"id":"reasoningEffort","value":"low"}]}],"result":{"error":"Primary provider cannot run"}}
FALLBACK_ABSENT {"calls":[{"instanceId":"codex","model":"gpt-6-luna","options":[{"id":"reasoningEffort","value":"low"}]}],"result":{"error":"Primary provider cannot run"}}
ACTUAL_CLI_MODELS [{"model":"claude-haiku-5-5"},{"model":"claude-haiku-5-5"},{"model":"claude-haiku-5-5"},{"model":"claude-haiku-5-5"},{"model":"z-ai/glm-5.3-flash"}]
DEFECT PRESENT: production runtime dispatched Haiku; custom-only backend failed; controls passed.

Screenshots, recordings, or supporting files

No response

Workaround

Set the text-generation model explicitly to the ready custom instance and model (claudeAgent / z-ai/glm-5.3-flash in this reproduction). This avoids the exhausted primary, and the reproduction shows it succeeding against the same custom-only fixture. The runtime fallback can still run, with the same problem, if that chosen provider itself becomes unusable.