ContextLab/orchestrator

Verify bare model-family resolution against the live API (blocked: no Anthropic credit)

Open

#432 opened on Jul 30, 2026

 (2 comments) (0 reactions) (0 assignees)Python (2 forks)auto 404
bughelp wanted

Repository metrics

Stars
 (3 stars)
PR merge metrics
 (PR metrics pending)

Description

Split out of the live-provider work so it is not silently forgotten.

What is verified

The live CI job confirmed against the real API:

test result
generate() with claude-haiku-4-5-20251001 PASSED
max_tokens actually applied PASSED
models.list() / discover_models() PASSED
health_check() PASSED

That validates the main fix: explicit model ids are no longer rewritten. Previously _normalize_model_name substring-matched the family and rewrote any id containing haiku/opus/sonnet to a hard-coded 2024 model, so every current Claude model was unreachable.

What is NOT verified

test_every_family_alias_resolves_to_a_real_model — whether a bare name ("haiku", "opus", "sonnet") resolves through client.models.list() to a servable id.

It could not run:

400 invalid_request_error: Your credit balance is too low to access the
Anthropic API.

This is an account precondition, not a known defect. The test now skips with that explicit reason rather than reporting a misleading failure, and still fails hard under ORCHESTRATOR_REQUIRE_LIVE=1 so the live job cannot go green on an unusable account.

To close

  1. Add credit to the Anthropic account.
  2. Run gh workflow run live-tests.yml.
  3. Confirm all five live tests pass. test_models_api_lists_servable_models prints the served model list, which is the primary source for which ids exist — two rounds of hard-coded ids have already been wrong here (retired 2024 dates, then invented -latest aliases).

Note on cost

The whole live suite is a handful of Haiku calls capped at max_tokens=32. The credit exhaustion was pre-existing, not caused by these tests.

Contributor guide