ContextLab/orchestrator

Verify bare model-family resolution against the live API (blocked: no Anthropic credit)

开放

#432 创建于 2026年7月30日

 (2 条评论) (0 个反应) (0 位负责人)Python (2 个派生)auto 404
bughelp wanted

仓库指标

星标
 (3 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Split out of the live-provider work so it is not silently forgotten.

What is verified

The live CI job confirmed against the real API:

test result
generate() with claude-haiku-4-5-20251001 PASSED
max_tokens actually applied PASSED
models.list() / discover_models() PASSED
health_check() PASSED

That validates the main fix: explicit model ids are no longer rewritten. Previously _normalize_model_name substring-matched the family and rewrote any id containing haiku/opus/sonnet to a hard-coded 2024 model, so every current Claude model was unreachable.

What is NOT verified

test_every_family_alias_resolves_to_a_real_model — whether a bare name ("haiku", "opus", "sonnet") resolves through client.models.list() to a servable id.

It could not run:

400 invalid_request_error: Your credit balance is too low to access the
Anthropic API.

This is an account precondition, not a known defect. The test now skips with that explicit reason rather than reporting a misleading failure, and still fails hard under ORCHESTRATOR_REQUIRE_LIVE=1 so the live job cannot go green on an unusable account.

To close

  1. Add credit to the Anthropic account.
  2. Run gh workflow run live-tests.yml.
  3. Confirm all five live tests pass. test_models_api_lists_servable_models prints the served model list, which is the primary source for which ids exist — two rounds of hard-coded ids have already been wrong here (retired 2024 dates, then invented -latest aliases).

Note on cost

The whole live suite is a handful of Haiku calls capped at max_tokens=32. The credit exhaustion was pre-existing, not caused by these tests.

贡献者指南