Unknown model ID silently falls back to another model without an error or model-change event

未關閉
  • 哪裡不對？
#2,856 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

評估

難度
4/5
預估耗時
3-5 天
新手友好度
48/100
Issue 類型
缺陷
描述清晰度
描述清楚
活躍度
活躍
技術堆疊
node.js, typescript
領域
backend-api-design

研究方向

Start by reading createSession() model selection and validation, then trace when the effective model is resolved and how session.warning and session.model_change events are emitted. Check existing model fallback tests and the related discussion in issue #250. Done means an unavailable explicitly requested model can be rejected or its fallback is observable, with behavior covered by tests and matching documentation.

由索引模型根據 Issue 內容生成。

描述

An explicitly requested, nonexistent model ID is accepted by createSession(). The first turn then runs on a different model without an exception, session.warning, or session.model_change event.

This prevents an SDK caller from relying on an explicit model selection. Checking session.rpc.model.getCurrent() immediately after creation does not detect the problem: it initially reports the invalid ID and only reports the fallback model after the turn.

This was reproduced on Copilot SDK 1.0.15 with runtime 1.0.89. The observation is based on SDK-reported per-API-call usage, not on asking the model to identify itself.

Environment

Component Version / configuration
SDK @github/copilot-sdk 1.0.15
Actual runtime, from client.getStatus() 1.0.89
Protocol version 3
Node.js 24.22.0
OS Windows
Mode copilot-cli
Authentication Authenticated Copilot client; no BYOK provider
Requested model invalid-model
Reasoning effort medium
Automatic routing Usage events reported isAuto: false
BYOK Usage events reported isByok: false
Observation date 2026-10-19

The SDK model catalog contained 16 models. invalid-model was absent. A valid control model, gpt-5.6-terra, was present and executed without substitution.

Steps to reproduce

Use an empty directory outside a source repository and an existing valid Copilot login. No repository files or external tools are needed. Ensure that no BYOK provider override is configured.

Install the observed SDK version:

npm init -y
npm install @github/[email protected] [email protected]

Save the following as repro.mjs, then run node repro.mjs.

Standalone SDK reproduction 
import { randomUUID } from 'node:crypto';
import { CopilotClient, defineTool } from '@github/copilot-sdk';
import { z } from 'zod';

const requestedModel = 'invalid-model';
const usage = [];
const notices = [];
const modelChanges = [];
let verdict;
const client = new CopilotClient({ mode: 'copilot-cli' });

try {
  await client.start();
  const models = await client.listModels();
  console.log({
    requestedModel,
    advertised: models.some((model) => model.id === requestedModel),
  });

  const session = await client.createSession({
    sessionId: randomUUID(),
    model: requestedModel,
    reasoningEffort: 'medium',
    systemMessage: {
      mode: 'replace',
      content: 'Call submit_verdict with passed set to true. Do not use other tools.',
    },
    tools: [defineTool('submit_verdict', {
      description: 'Submit the probe verdict.',
      parameters: z.object({ passed: z.boolean() }),
      skipPermission: true,
      handler(args) {
        verdict = args;
        return 'Grade submitted. Do not respond further.';
      },
    })],
    availableTools: ['submit_verdict'],
    onPermissionRequest: () => ({
      kind: 'reject',
      feedback: 'The probe denies other tool and file access.',
    }),
    infiniteSessions: { enabled: false },
    enableSessionStore: false,
    enableSessionTelemetry: false,
    skipCustomInstructions: true,
    enableConfigDiscovery: false,
    enableSkills: false,
    enableFileHooks: false,
    onEvent(event) {
      if (event.type === 'assistant.usage') {
        const { model, isAuto, isByok } = event.data;
        usage.push({ model, isAuto, isByok });
      }
      if (event.type === 'session.warning' || event.type === 'session.error') {
        notices.push({ type: event.type, message: event.data.message });
      }
      if (event.type === 'session.model_change') {
        modelChanges.push(event.data);
      }
    },
  });

  const selectedAfterCreate = (await session.rpc.model.getCurrent()).modelId;
  await session.sendAndWait({ prompt: 'Submit the verdict now.' }, 45_000);
  console.log(JSON.stringify({
    requestedModel,
    selectedAfterCreate,
    selectedAfterTurn: (await session.rpc.model.getCurrent()).modelId,
    usage,
    notices,
    modelChanges,
    verdict,
  }, null, 2));
} finally {
  const errors = await client.stop();
  if (errors.length) throw new AggregateError(errors, 'SDK cleanup failed.');
}

The observer is provided in createSession({ onEvent }), with an explicit session ID, so it is installed before session creation completes. The absence of a warning is not explained by attaching an observer after the turn.

For a control, replace invalid-model with a model ID returned by listModels() for the account.

Expected behavior

For applications that explicitly pin a model, there should be a supported way to reject an unavailable model rather than execute on a different one. A strict-model / no-fallback option would address this without necessarily changing the default behavior for existing applications.

If fallback remains enabled, it should be observable:

  • Emit a warning and/or a model-change event identifying the requested and effective models and the reason.
  • Document when the effective model is resolved and why the initial getCurrent() value may not represent the model that will execute.
  • Make it possible to enforce the requested model without relying on a successful inference call on the wrong model.

Actual behavior and captured evidence

The live capture produced the following relevant fields:

{
  "requestedModel": "invalid-model",
  "sdkRequestModel": "invalid-model",
  "selectedAfterCreate": "invalid-model",
  "selectedAfterTurn": "claude-sonnet-5",
  "usage": [
    { "model": "claude-sonnet-5", "isAuto": false, "isByok": false },
    { "model": "claude-sonnet-5", "isAuto": false, "isByok": false }
  ],
  "notices": [],
  "modelChanges": [],
  "verdict": { "passed": true }
}

listModels() did not advertise invalid-model. Nevertheless, both API calls used claude-sonnet-5, the tool submitted a successful verdict, and the SDK did not report a session warning, session error, or model-change event.

claude-sonnet-5 is the fallback observed in this environment for this particular execution, not a claim about a universal or hard-coded default.

Impact

  • Evaluations can execute on a different judge model while appearing to honor an explicit configuration.
  • A typo or removed model ID can change model behavior, cost, and reproducibility without a configuration error.
  • Inspecting only the initial selected model is insufficient to enforce model identity.
  • Consumers may incorrectly attribute usage to the requested ID unless they independently capture per-call usage.

Related reports and documentation

主要語言
Java
星號
10.7k
分支
1.5k
30 天內沒有已合併 PR

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 沒有 Pull Request 範本
  • 閱讀貢獻指南

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

github/copilot-sdk 的其他 Issue

查看 github/copilot-sdk 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。