bug(models): Refresh Models reports success without invalidating live-model caches

Summary

The Settings → Providers → Refresh Models action can report success without refreshing the live model catalog for a provider.

This is especially visible with an OpenAI-compatible custom provider whose /v1/models catalog gains a new model while the WebUI process and browser tab remain open. The new model does not appear until a full page reload (and, depending on the server cache state, until the live cache is also refreshed).

Ownership: WebUI, not Hermes Agent

This appears to be a WebUI cache-invalidation bug rather than a Hermes Agent model-discovery bug:

• POST /api/models/refresh is implemented in nesquena/hermes-webui.
• The server-side /api/models/live TTL cache is implemented in api/routes.py in this repository.
• The browser-side _liveModelCache is implemented in static/ui.js in this repository.
• The stale-cache check happens before _handle_live_models() delegates to hermes_cli.models.provider_model_ids() or runs the WebUI custom-provider fallback fetch.

Therefore, when the stale WebUI cache hits, the Hermes Agent discovery path is never reached.

Reproduction

A provider can be any OpenAI-compatible custom endpoint; no provider-specific behavior is required.

1. Configure a provider such as custom:example-relay whose /v1/models endpoint initially returns model-before-refresh.
2. Open the WebUI model picker or Settings → Providers so the live model response is cached.
3. Change the upstream /v1/models response so that it returns a new model-after-refresh.
4. Click Settings → Providers → Refresh Models for that provider.
5. Observe the success toast.
6. Re-open the model picker or inspect /api/models/live?provider=custom%3Aexample-relay.

Actual behavior

The old live response is still returned. The new model is absent. A full browser page reload causes a new browser-side live-model request and then the new model appears.

Expected behavior

The Refresh Models action should invalidate the relevant server-side and browser-side live-model caches, perform a fresh live-model request, and update all model-picker surfaces before reporting success.

Deterministic current-master reproduction

I ran a minimal isolated harness against origin/master at:

bf9b61ff3dc35d19a9428670cd7ff9288752d239

The harness returned one model on the first live lookup, changed the provider response, then performed the exact backend cache invalidation used by POST /api/models/refresh:

first_model                         = provider/model-before-refresh
after_refresh_endpoint_side_effect  = provider/model-before-refresh
after_explicit_live_cache_clear     = provider/model-after-refresh
provider_fetch_calls                = 2
bug_reproduced                       = true

This demonstrates that invalidate_provider_models_cache(provider_id) does not invalidate the separate live-model cache used by /api/models/live.

A real OpenAI-compatible provider was also verified to return HTTP 200 from /v1/models with the newly added model present. The provider URL and credentials are intentionally omitted from this public report.

Source trace

1. Refresh route only invalidates the primary model catalog

Current master implements the button route as:

• api/routes.py#L14762-L14768
• It calls invalidate_provider_models_cache().

That invalidator clears the /api/models catalog cache and disk cache, but it does not clear the separate _LIVE_MODELS_CACHE in api/routes.py.

2. /api/models/live returns the stale server cache before discovery

The live cache is defined and checked here:

• api/routes.py#L1863-L1883
• api/routes.py#L20176-L20199

The handler returns the cached payload before it reaches the Hermes Agent delegation or the WebUI custom-provider fallback path.

3. The browser cache is not cleared by Refresh Models

The frontend keeps a process-lifetime cache:

• static/ui.js#L3525-L3526
• static/ui.js#L3631-L3652

_fetchLiveModels() returns _liveModelCache[provider] immediately when present, without requesting /api/models/live again.

The Refresh Models success path only calls _refreshModelDropdownsAfterProviderChange():

• static/panels.js#L11767-L11794

That helper invalidates the slash-model cache and resets model-dropdown readiness, but it does not invalidate _liveModelCache[provider] or force a fresh /api/models/live request.

Important distinction from the allowlist behavior

A non-empty custom_providers[].models list is intentionally treated as an allowlist. The relevant upstream change explicitly says configured models take priority over live /v1/models discovery:

• 9c3e37d2e

That behavior is useful for large aggregator catalogs and is not the bug reported here. The issue is that the explicit Refresh Models action does not invalidate the live discovery caches that are supposed to let a user inspect newly available models before deciding whether to add them to an allowlist.

This is also distinct from the already-closed #3718, which fixed an older bug where a custom provider with a configured model: entry skipped the live probe entirely. The current problem is that the live probe can be correctly implemented but still bypassed by stale WebUI caches.

The larger interactive custom-provider model manager in PR #4852 is related product work, but this report is intentionally limited to the refresh/cache contract and should be fixable independently.

Suggested fix direction

Please consider:

1. Adding provider-scoped invalidation for the /api/models/live server cache as part of POST /api/models/refresh.
2. Clearing the corresponding frontend _liveModelCache[provider] entry in the Refresh Models success path.
3. Ensuring the refresh path performs or awaits one fresh live fetch before showing the success toast, so “refreshed” means that the live catalog was actually re-read.
4. Adding a regression test covering:
   • a warmed live cache;
   • a changed upstream model response;
   • POST /api/models/refresh;
   • a subsequent /api/models/live request returning the new model;
   • frontend cache invalidation for the same provider.

Environment

• Repository: nesquena/hermes-webui
• Verified against: origin/master bf9b61ff3dc35d19a9428670cd7ff9288752d239
• Latest release observed during investigation: v0.52.106
• Provider shape: OpenAI-compatible custom endpoint with /v1/models
• Hermes Agent installed alongside the WebUI: v0.20.0

No credentials, private provider URLs, local filesystem paths, session IDs, or raw logs are included in this report.