Unknown model ID silently falls back to another model without an error or model-change event

Aberta
  • Algo errado?
#2,856 0 comentários 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
48/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
node.js, typescript
Domínio
backend-api-design

Direção de pesquisa

Start by reading createSession() model selection and validation, then trace when the effective model is resolved and how session.warning and session.model_change events are emitted. Check existing model fallback tests and the related discussion in issue #250. Done means an unavailable explicitly requested model can be rejected or its fallback is observable, with behavior covered by tests and matching documentation.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

An explicitly requested, nonexistent model ID is accepted by createSession(). The first turn then runs on a different model without an exception, session.warning, or session.model_change event.

This prevents an SDK caller from relying on an explicit model selection. Checking session.rpc.model.getCurrent() immediately after creation does not detect the problem: it initially reports the invalid ID and only reports the fallback model after the turn.

This was reproduced on Copilot SDK 1.0.15 with runtime 1.0.89. The observation is based on SDK-reported per-API-call usage, not on asking the model to identify itself.

Environment

Component Version / configuration
SDK @github/copilot-sdk 1.0.15
Actual runtime, from client.getStatus() 1.0.89
Protocol version 3
Node.js 24.22.0
OS Windows
Mode copilot-cli
Authentication Authenticated Copilot client; no BYOK provider
Requested model invalid-model
Reasoning effort medium
Automatic routing Usage events reported isAuto: false
BYOK Usage events reported isByok: false
Observation date 2026-10-19

The SDK model catalog contained 16 models. invalid-model was absent. A valid control model, gpt-5.6-terra, was present and executed without substitution.

Steps to reproduce

Use an empty directory outside a source repository and an existing valid Copilot login. No repository files or external tools are needed. Ensure that no BYOK provider override is configured.

Install the observed SDK version:

npm init -y
npm install @github/[email protected] [email protected]

Save the following as repro.mjs, then run node repro.mjs.

Standalone SDK reproduction 
import { randomUUID } from 'node:crypto';
import { CopilotClient, defineTool } from '@github/copilot-sdk';
import { z } from 'zod';

const requestedModel = 'invalid-model';
const usage = [];
const notices = [];
const modelChanges = [];
let verdict;
const client = new CopilotClient({ mode: 'copilot-cli' });

try {
  await client.start();
  const models = await client.listModels();
  console.log({
    requestedModel,
    advertised: models.some((model) => model.id === requestedModel),
  });

  const session = await client.createSession({
    sessionId: randomUUID(),
    model: requestedModel,
    reasoningEffort: 'medium',
    systemMessage: {
      mode: 'replace',
      content: 'Call submit_verdict with passed set to true. Do not use other tools.',
    },
    tools: [defineTool('submit_verdict', {
      description: 'Submit the probe verdict.',
      parameters: z.object({ passed: z.boolean() }),
      skipPermission: true,
      handler(args) {
        verdict = args;
        return 'Grade submitted. Do not respond further.';
      },
    })],
    availableTools: ['submit_verdict'],
    onPermissionRequest: () => ({
      kind: 'reject',
      feedback: 'The probe denies other tool and file access.',
    }),
    infiniteSessions: { enabled: false },
    enableSessionStore: false,
    enableSessionTelemetry: false,
    skipCustomInstructions: true,
    enableConfigDiscovery: false,
    enableSkills: false,
    enableFileHooks: false,
    onEvent(event) {
      if (event.type === 'assistant.usage') {
        const { model, isAuto, isByok } = event.data;
        usage.push({ model, isAuto, isByok });
      }
      if (event.type === 'session.warning' || event.type === 'session.error') {
        notices.push({ type: event.type, message: event.data.message });
      }
      if (event.type === 'session.model_change') {
        modelChanges.push(event.data);
      }
    },
  });

  const selectedAfterCreate = (await session.rpc.model.getCurrent()).modelId;
  await session.sendAndWait({ prompt: 'Submit the verdict now.' }, 45_000);
  console.log(JSON.stringify({
    requestedModel,
    selectedAfterCreate,
    selectedAfterTurn: (await session.rpc.model.getCurrent()).modelId,
    usage,
    notices,
    modelChanges,
    verdict,
  }, null, 2));
} finally {
  const errors = await client.stop();
  if (errors.length) throw new AggregateError(errors, 'SDK cleanup failed.');
}

The observer is provided in createSession({ onEvent }), with an explicit session ID, so it is installed before session creation completes. The absence of a warning is not explained by attaching an observer after the turn.

For a control, replace invalid-model with a model ID returned by listModels() for the account.

Expected behavior

For applications that explicitly pin a model, there should be a supported way to reject an unavailable model rather than execute on a different one. A strict-model / no-fallback option would address this without necessarily changing the default behavior for existing applications.

If fallback remains enabled, it should be observable:

  • Emit a warning and/or a model-change event identifying the requested and effective models and the reason.
  • Document when the effective model is resolved and why the initial getCurrent() value may not represent the model that will execute.
  • Make it possible to enforce the requested model without relying on a successful inference call on the wrong model.

Actual behavior and captured evidence

The live capture produced the following relevant fields:

{
  "requestedModel": "invalid-model",
  "sdkRequestModel": "invalid-model",
  "selectedAfterCreate": "invalid-model",
  "selectedAfterTurn": "claude-sonnet-5",
  "usage": [
    { "model": "claude-sonnet-5", "isAuto": false, "isByok": false },
    { "model": "claude-sonnet-5", "isAuto": false, "isByok": false }
  ],
  "notices": [],
  "modelChanges": [],
  "verdict": { "passed": true }
}

listModels() did not advertise invalid-model. Nevertheless, both API calls used claude-sonnet-5, the tool submitted a successful verdict, and the SDK did not report a session warning, session error, or model-change event.

claude-sonnet-5 is the fallback observed in this environment for this particular execution, not a claim about a universal or hard-coded default.

Impact

  • Evaluations can execute on a different judge model while appearing to honor an explicit configuration.
  • A typo or removed model ID can change model behavior, cost, and reproducibility without a configuration error.
  • Inspecting only the initial selected model is insufficient to enforce model identity.
  • Consumers may incorrectly attribute usage to the requested ID unless they independently capture per-call usage.

Related reports and documentation

Linguagem predominante
Java
Estrelas
10.7k
Forks
1.5k
Nenhum PR com merge em 30d

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de github/copilot-sdk

Todas as issues de github/copilot-sdk

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.