[Feature]: Discover available models from OpenAI-compatible custom base_url for the model picker and drift validation

Open
#7,005 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
43/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
openapi, python

Research direction

Start by reading the agent-server route and profile validation code, then inspect the server-side fetch patterns in runtime_metadata.py and openrouter.py. Add the upstream catalog endpoint with normalized model IDs and warning-only validation; confirm both response shapes, auth handling, redacted failures, and the OpenAPI and TypeScript client updates. Done means the listed acceptance criteria pass, including the local HTTP stand-in tests.

Written by the indexing model from the issue text.

Description

priority:low ready-for-dev

Summary

Add a way for the model picker / LLM profile editor to discover available models from an OpenAI-compatible custom base_url (fetch GET {base_url}/v1/models), instead of only offering LiteLLM's static model list (GET /api/llm/models) and the hardcoded verified lists.

Problem

When an LLM profile points at a local proxy or gateway (base_url set, e.g. LiteLLM proxy, one-api/new-api style aggregators, or a self-hosted router), the set of valid model IDs is determined by that upstream, not by LiteLLM's static catalog. Today:

  1. The settings UI model picker is populated from GET /api/llm/models (LiteLLM static list, ~3.8k entries) plus GET /api/llm/models/verified. Models that exist only behind the custom endpoint are not listed; conversely, listed names may not exist upstream.
  2. If the saved profile model references an upstream-only ID, nothing validates it against the upstream catalog. A mismatch surfaces only at conversation time as an upstream 400 model_not_found, which is non-retryable and kills the conversation (LLMBadRequestError, classification.kind=config).

Concrete incident from our deployment: a profile pinned to one upstream's model ID kept working while a router in front of it redirected traffic to a different provider with a strict pass-through model mapping — the stale model ID then failed every new conversation with 400 model_not_found until manually re-pointed. A catalog check against base_url would have detected the drift immediately.

Related work

  • #6863 / #6862 — provider connections becoming authoritative for routing and model ID normalization. This issue complements that work on the discovery/validation side: knowing which IDs the endpoint actually serves.
  • The SDK already live-fetches metadata for one special-cased provider (OpenRouter, hardcoded openrouter.ai endpoint in sdk/llm/utils/providers/openrouter.py). This request generalizes that pattern to any OpenAI-compatible base_url.
  • LiteLLM itself exposes client.models.list() for OpenAI-compatible endpoints, so the underlying client capability exists.

Proposal

  1. Catalog endpoint on the agent-server, e.g. GET /api/llm/models/upstream?base_url=... (or accept a profile name): fetch {base_url}/v1/models server-side, normalize both OpenAI format ({"data":[{"id":...}]}) and common variants, return {models: [...]}. No secrets leave the server; the profile's stored API key is used for authenticated endpoints, with an anonymous fallback for open endpoints.
  2. Model picker integration: when a profile has a base_url, offer "models from this endpoint" (live fetch) alongside the static LiteLLM list, with a clear source label.
  3. Optional drift check: when saving/activating a profile, optionally validate model against the fetched catalog (warning, not a hard error — some gateways lazily accept IDs).

Notes on format variance

Real-world gateways vary: some return strict OpenAI data[] format, others return flat arrays like {"models":[{"slug":...,"display_name":...}]} (the OpenAI Codex CLI native catalog format). Accepting both (and maybe id/slug/name keys) covers most deployments; a normalization helper would also let proxy projects add an OpenAI-compatible models endpoint cheaply.

Environment

  • OpenHands Agent Server 1.62.0 / SDK 1.62.0 (WSL2, local Agent Canvas)
  • Proxy in front: third-party LLM router providing OpenAI-compatible /v1/responses + native /_v1/models

This issue was created by an AI agent (OpenHands) on behalf of @JohnieXu.


OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

Bounded scope: this repository owns the Python SDK and the agent-server REST API (AGENTS.md), so the discovery endpoint and the profile-validation hook belong here. Add one additive agent-server endpoint, GET /api/llm/models/upstream, that takes a base_url (or a profile name) server-side, fetches that endpoint's model catalog with httpx (same server-side fetch pattern as runtime_metadata.py/openrouter.py), normalizes both OpenAI data[].id and flat models[] (id/slug/name) shapes, and returns {models: [...]}. The request uses the stored profile/provider-connection API key server-side when one exists, with an anonymous fallback; no secrets leave the server and none may appear in responses, logs, or errors. POST /api/llm/profiles/{name}/validate may add an optional, non-blocking catalog check of the profile's model against the fetched catalog (warning-only, mirroring _TRANSIENT_ERROR_TYPES treatment). Model-picker UI presentation is a Canvas change (OpenHands/OpenHands); the endpoint must land here first per the SDK → OpenAPI → clients/typescript → Canvas flow.

Non-goals: no change to GET /api/llm/models, GET /api/llm/models/verified, or VERIFIED_MODELS/unverified_models.py catalogs; no new LiteLLM provider integration; no hard blocking on save/activation for unknown model IDs (some gateways lazily accept IDs); no rewrite of stored profiles; no Canvas UI work in this repository.

Acceptance Criteria
  • GET /api/llm/models/upstream accepts a base_url query parameter (or a stored profile name) and returns HTTP 200 with {"models": [...]} listing the model IDs the upstream endpoint serves, for both OpenAI {"data":[{"id":...}]} and flat {"models":[...]} response shapes (normalizing id/slug/name keys).
  • When the profile or its linked provider connection has an API key, the upstream catalog request is sent server-side with that key; the key never appears in the response body, logs, or error text.
  • Anonymous endpoints reachable without credentials are fetched without an API key instead of failing.
  • The upstream request is bounded by a finite timeout; timeout, DNS failure, non-2xx status, and unparseable bodies return a typed 4xx/5xx error (not a hang and not a raw 500) with a redacted message.
  • POST /api/llm/profiles/{name}/validate optionally reports a catalog mismatch between the profile's model and the fetched upstream catalog as a non-blocking warning; a failed catalog fetch alone does not fail validation.
  • The change is additive: no existing /api/** operation or response schema changes (the agent-server REST breakage check passes), and GET /api/llm/models plus /models/verified behavior is unchanged.
  • The endpoint appears in the agent-server OpenAPI schema and is mirrored in clients/typescript/ per the SDK → OpenAPI → TypeScript client flow.
  • Automated tests cover the route against a local HTTP stand-in: both response shapes, authenticated and anonymous fetch, timeout/non-2xx/malformed-body errors, and the warning-only validate behavior; live coverage where credentials are available is not required.
Dominant language
Python
Stars
1.4k
Forks
797
Avg merge
1d 17h
Merged PRs (30d)
354

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenHands/software-agent-sdk

All issues in OpenHands/software-agent-sdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.