Verify bare model-family resolution against the live API (blocked: no Anthropic credit)
#432 aberto em 30 de jul. de 2026
Métricas do repositório
- Stars
- (3 estrelas)
- Métricas de merge de PR
- (Métricas PR pendentes)
Description
Split out of the live-provider work so it is not silently forgotten.
What is verified
The live CI job confirmed against the real API:
| test | result |
|---|---|
generate() with claude-haiku-4-5-20251001 |
PASSED |
max_tokens actually applied |
PASSED |
models.list() / discover_models() |
PASSED |
health_check() |
PASSED |
That validates the main fix: explicit model ids are no longer rewritten. Previously _normalize_model_name substring-matched the family and rewrote any id containing haiku/opus/sonnet to a hard-coded 2024 model, so every current Claude model was unreachable.
What is NOT verified
test_every_family_alias_resolves_to_a_real_model — whether a bare name ("haiku", "opus", "sonnet") resolves through client.models.list() to a servable id.
It could not run:
400 invalid_request_error: Your credit balance is too low to access the
Anthropic API.
This is an account precondition, not a known defect. The test now skips with that explicit reason rather than reporting a misleading failure, and still fails hard under ORCHESTRATOR_REQUIRE_LIVE=1 so the live job cannot go green on an unusable account.
To close
- Add credit to the Anthropic account.
- Run
gh workflow run live-tests.yml. - Confirm all five live tests pass.
test_models_api_lists_servable_modelsprints the served model list, which is the primary source for which ids exist — two rounds of hard-coded ids have already been wrong here (retired 2024 dates, then invented-latestaliases).
Note on cost
The whole live suite is a handful of Haiku calls capped at max_tokens=32. The credit exhaustion was pre-existing, not caused by these tests.