Unknown model ID silently falls back to another model without an error or model-change event

Đang mở
  • Có gì không ổn?
#2,856 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
node.js, typescript
Lĩnh vực
backend-api-design

Hướng nghiên cứu

Start by reading createSession() model selection and validation, then trace when the effective model is resolved and how session.warning and session.model_change events are emitted. Check existing model fallback tests and the related discussion in issue #250. Done means an unavailable explicitly requested model can be rejected or its fallback is observable, with behavior covered by tests and matching documentation.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

An explicitly requested, nonexistent model ID is accepted by createSession(). The first turn then runs on a different model without an exception, session.warning, or session.model_change event.

This prevents an SDK caller from relying on an explicit model selection. Checking session.rpc.model.getCurrent() immediately after creation does not detect the problem: it initially reports the invalid ID and only reports the fallback model after the turn.

This was reproduced on Copilot SDK 1.0.15 with runtime 1.0.89. The observation is based on SDK-reported per-API-call usage, not on asking the model to identify itself.

Environment

Component Version / configuration
SDK @github/copilot-sdk 1.0.15
Actual runtime, from client.getStatus() 1.0.89
Protocol version 3
Node.js 24.22.0
OS Windows
Mode copilot-cli
Authentication Authenticated Copilot client; no BYOK provider
Requested model invalid-model
Reasoning effort medium
Automatic routing Usage events reported isAuto: false
BYOK Usage events reported isByok: false
Observation date 2026-10-19

The SDK model catalog contained 16 models. invalid-model was absent. A valid control model, gpt-5.6-terra, was present and executed without substitution.

Steps to reproduce

Use an empty directory outside a source repository and an existing valid Copilot login. No repository files or external tools are needed. Ensure that no BYOK provider override is configured.

Install the observed SDK version:

npm init -y
npm install @github/[email protected] [email protected]

Save the following as repro.mjs, then run node repro.mjs.

Standalone SDK reproduction 
import { randomUUID } from 'node:crypto';
import { CopilotClient, defineTool } from '@github/copilot-sdk';
import { z } from 'zod';

const requestedModel = 'invalid-model';
const usage = [];
const notices = [];
const modelChanges = [];
let verdict;
const client = new CopilotClient({ mode: 'copilot-cli' });

try {
  await client.start();
  const models = await client.listModels();
  console.log({
    requestedModel,
    advertised: models.some((model) => model.id === requestedModel),
  });

  const session = await client.createSession({
    sessionId: randomUUID(),
    model: requestedModel,
    reasoningEffort: 'medium',
    systemMessage: {
      mode: 'replace',
      content: 'Call submit_verdict with passed set to true. Do not use other tools.',
    },
    tools: [defineTool('submit_verdict', {
      description: 'Submit the probe verdict.',
      parameters: z.object({ passed: z.boolean() }),
      skipPermission: true,
      handler(args) {
        verdict = args;
        return 'Grade submitted. Do not respond further.';
      },
    })],
    availableTools: ['submit_verdict'],
    onPermissionRequest: () => ({
      kind: 'reject',
      feedback: 'The probe denies other tool and file access.',
    }),
    infiniteSessions: { enabled: false },
    enableSessionStore: false,
    enableSessionTelemetry: false,
    skipCustomInstructions: true,
    enableConfigDiscovery: false,
    enableSkills: false,
    enableFileHooks: false,
    onEvent(event) {
      if (event.type === 'assistant.usage') {
        const { model, isAuto, isByok } = event.data;
        usage.push({ model, isAuto, isByok });
      }
      if (event.type === 'session.warning' || event.type === 'session.error') {
        notices.push({ type: event.type, message: event.data.message });
      }
      if (event.type === 'session.model_change') {
        modelChanges.push(event.data);
      }
    },
  });

  const selectedAfterCreate = (await session.rpc.model.getCurrent()).modelId;
  await session.sendAndWait({ prompt: 'Submit the verdict now.' }, 45_000);
  console.log(JSON.stringify({
    requestedModel,
    selectedAfterCreate,
    selectedAfterTurn: (await session.rpc.model.getCurrent()).modelId,
    usage,
    notices,
    modelChanges,
    verdict,
  }, null, 2));
} finally {
  const errors = await client.stop();
  if (errors.length) throw new AggregateError(errors, 'SDK cleanup failed.');
}

The observer is provided in createSession({ onEvent }), with an explicit session ID, so it is installed before session creation completes. The absence of a warning is not explained by attaching an observer after the turn.

For a control, replace invalid-model with a model ID returned by listModels() for the account.

Expected behavior

For applications that explicitly pin a model, there should be a supported way to reject an unavailable model rather than execute on a different one. A strict-model / no-fallback option would address this without necessarily changing the default behavior for existing applications.

If fallback remains enabled, it should be observable:

  • Emit a warning and/or a model-change event identifying the requested and effective models and the reason.
  • Document when the effective model is resolved and why the initial getCurrent() value may not represent the model that will execute.
  • Make it possible to enforce the requested model without relying on a successful inference call on the wrong model.

Actual behavior and captured evidence

The live capture produced the following relevant fields:

{
  "requestedModel": "invalid-model",
  "sdkRequestModel": "invalid-model",
  "selectedAfterCreate": "invalid-model",
  "selectedAfterTurn": "claude-sonnet-5",
  "usage": [
    { "model": "claude-sonnet-5", "isAuto": false, "isByok": false },
    { "model": "claude-sonnet-5", "isAuto": false, "isByok": false }
  ],
  "notices": [],
  "modelChanges": [],
  "verdict": { "passed": true }
}

listModels() did not advertise invalid-model. Nevertheless, both API calls used claude-sonnet-5, the tool submitted a successful verdict, and the SDK did not report a session warning, session error, or model-change event.

claude-sonnet-5 is the fallback observed in this environment for this particular execution, not a claim about a universal or hard-coded default.

Impact

  • Evaluations can execute on a different judge model while appearing to honor an explicit configuration.
  • A typo or removed model ID can change model behavior, cost, and reproducibility without a configuration error.
  • Inspecting only the initial selected model is insufficient to enforce model identity.
  • Consumers may incorrectly attribute usage to the requested ID unless they independently capture per-call usage.

Related reports and documentation

Ngôn ngữ chính
Java
Star
10.7k
Fork
1.5k
Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của github/copilot-sdk

Tất cả issue của github/copilot-sdk

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.