Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

chore(scripts): model sync emits `tools: []` and skips ANTHROPIC_COMBINED_TOOLS_AND_SCHEMA_MODELS, so every new Anthropic model lands red (#1516)

Open
  • Is something wrong?
#1,573 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
ai, testing-qa, tooling

Research direction

Start with scripts/model-sync/provider-supports.ts and scripts/sync-provider-models.ts, then run the failing Anthropic and OpenRouter tests named in the issue. Check the generated metadata against the hand-maintained Anthropic capability list and the catalog-based OpenRouter test. Done means the sync regenerates green metadata and the Sync Model Metadata workflow no longer fails.

Written by the indexing model from the issue text.

Description

Problem

The daily model sync (automated/sync-models, PR #1516) is red on Test and Coverage. The Coverage job is not reporting a coverage drop — @tanstack/ai-anthropic:test:coverage errors out, so the job exits 1 before any comparison runs.

Run: https://github.com/TanStack/ai/actions/runs/36563236332 (head 3cbc3a061)

Three failures, all from the newly synced claude-sonnet-5-5 plus one removed OpenRouter model. All three reproduce locally:

  1. @tanstack/ai-anthropic:test:lib — tests/model-meta.test.ts:418, "covers every registered model that accepts output_config.format":

    AssertionError: expected [ 'claude-sonnet-5-5' ] to deeply equal []

    The sync added CLAUDE_SONNET_5_5.id to ANTHROPIC_MODELS and to the three type maps, but not to ANTHROPIC_COMBINED_TOOLS_AND_SCHEMA_MODELS (packages/ai-anthropic/src/model-meta.ts:778).

  2. @tanstack/ai-anthropic:test:types — tests/tools-per-model-type-safety.test.ts:252, "every registered model declares at least one provider tool":

    error TS2344: Type '"claude-opus-5-fast"' does not satisfy the constraint
'"Expected: literal string: claude-opus-5-fast, Actual: literal string: claude-sonnet-5-5"'

    The generated entry has supports: { tools: [] }, so the new model joins claude-opus-5-fast in the "declares no provider tool" set that the test pins to exactly one id. For comparison, the hand-written CLAUDE_SONNET_5 (packages/ai-anthropic/src/model-meta.ts:464) lists web_search, web_fetch, code_execution, computer_use, bash, text_editor, memory. The generated entry also omits the extended_thinking / adaptive_thinking / priority_tier flags its siblings carry — same root cause, not caught by any test.

  3. @tanstack/ai-openrouter:test:types — tests/openrouter-combined-structured-output.test.ts:187: 'anthropic/claude-3-haiku' is no longer in OPENROUTER_CHAT_MODELS. This sync dropped it (along with deepseek/deepseek-r1-distill-llama-70b and four :free variants), so the literal no longer type-checks.

Root cause

1 and 2 are generator bugs and recur on every sync that adds a native Anthropic model.

scripts/model-sync/provider-supports.ts hardcodes tools: [] for every provider (case 'anthropic' at line 118, and the same literal in the openai, gemini and grok branches) — a conservative placeholder that was never filled in. scripts/sync-provider-models.ts:443 emits it verbatim, and nothing in the sync knows about ANTHROPIC_COMBINED_TOOLS_AND_SCHEMA_MODELS, so a new model lands registered-but-unlisted.

Same class as #1163: the sync writes a generated model list while a hand-maintained companion list goes stale.

3 is unrelated test brittleness — the test pins a literal catalog id that OpenRouter can (and did) delete.

Fix

  • scripts/model-sync/provider-supports.ts: fill the real provider server tools for new models instead of emitting tools: [], per provider (Anthropic, OpenAI, Gemini, Grok each have their own vocabulary — Anthropic web_search/code_execution/…, OpenAI web_search/file_search/code_interpreter/mcp/…, Gemini code_execution/google_search/url_context/…). While in there, emit the per-generation Anthropic flags (extended_thinking / adaptive_thinking / priority_tier) the hand-written entries carry.
  • scripts/sync-provider-models.ts: insert a new native Anthropic model id into ANTHROPIC_COMBINED_TOOLS_AND_SCHEMA_MODELS when it accepts output_config.format.
  • packages/ai-openrouter/tests/openrouter-combined-structured-output.test.ts:187: stop pinning a literal that a sync can delete — pick the model from the catalog by capability instead.
  • Then re-run Sync Model Metadata (workflow_dispatch) so #1516 regenerates green. Fixing #1516's branch directly is pointless: .github/workflows/sync-models.yml:52 rebuilds it from main and git push --forces it every day at 06:00 UTC.
Dominant language
TypeScript
Stars
3.1k
Forks
340
Avg merge
2d 7h
Merged PRs (30d)
175

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from TanStack/ai

All issues in TanStack/ai

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.