ContextLab/orchestrator

Verify bare model-family resolution against the live API (blocked: no Anthropic credit)

オープン

#432 opened on 2026/07/30

 (2 件のコメント) (0 件のリアクション) (0 人の担当者)Python (2 件のフォーク)auto 404
bughelp wanted

Repository metrics

Stars
 (3 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Split out of the live-provider work so it is not silently forgotten.

What is verified

The live CI job confirmed against the real API:

test result
generate() with claude-haiku-4-5-20251001 PASSED
max_tokens actually applied PASSED
models.list() / discover_models() PASSED
health_check() PASSED

That validates the main fix: explicit model ids are no longer rewritten. Previously _normalize_model_name substring-matched the family and rewrote any id containing haiku/opus/sonnet to a hard-coded 2024 model, so every current Claude model was unreachable.

What is NOT verified

test_every_family_alias_resolves_to_a_real_model — whether a bare name ("haiku", "opus", "sonnet") resolves through client.models.list() to a servable id.

It could not run:

400 invalid_request_error: Your credit balance is too low to access the
Anthropic API.

This is an account precondition, not a known defect. The test now skips with that explicit reason rather than reporting a misleading failure, and still fails hard under ORCHESTRATOR_REQUIRE_LIVE=1 so the live job cannot go green on an unusable account.

To close

  1. Add credit to the Anthropic account.
  2. Run gh workflow run live-tests.yml.
  3. Confirm all five live tests pass. test_models_api_lists_servable_models prints the served model list, which is the primary source for which ids exist — two rounds of hard-coded ids have already been wrong here (retired 2024 dates, then invented -latest aliases).

Note on cost

The whole live suite is a handful of Haiku calls capped at max_tokens=32. The credit exhaustion was pre-existing, not caused by these tests.

コントリビューターガイド