feat: support client-driven model selection in ai-proxy-multi via `models` request body field

Abierto
  • ¿Algo no está bien?
#13,083 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

  • #13084 de @kjprice — cerrado sin fusionar

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
45/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
lua
Área
api, backend-api-design

Línea de trabajo

Comienza leyendo ai-proxy-multi.lua y su flujo de selección de instancias en la access-phase. Traza cómo se gestionan los cuerpos de las solicitudes y las instancias configuradas antes del reenvío al upstream, y luego define pruebas para el orden opt-in, la coincidencia de cadenas y objetos, las entradas sin coincidencia, la eliminación de modelos y el comportamiento de fallback; se considera terminado cuando el comportamiento predeterminado sigue estando controlado por el servidor, mientras que las rutas habilitadas respetan las preferencias del cliente.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

feature-request

Description

Currently, ai-proxy-multi selects AI instances entirely server-side via configured priority, weight, and fallback_strategy. The client has no way to influence which model or provider handles their request, or in what order fallbacks should be attempted.

This proposal adds support for a models field in the request body, allowing clients to specify their preferred model ordering while the gateway retains full control over authentication, rate limiting, and provider configuration.

Proposed API

Clients include a models array in the chat completions request body. Each element can be:

Object form (full control):

{
  "messages": [{"role": "user", "content": "Hello"}],
  "models": [
    {"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
    {"provider": "openai", "model": "gpt-4o"}
  ]
}

String shorthand (just model names, matched against configured instances):

{
  "messages": [{"role": "user", "content": "Hello"}],
  "models": ["claude-sonnet-4-20250514", "gpt-4o"]
}

Mixed:

{
  "messages": [{"role": "user", "content": "Hello"}],
  "models": [
    {"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
    "gpt-4o"
  ]
}

Behavior

  1. The models array defines the client's preferred instance ordering (first element = highest priority).
  2. Each entry is matched against the route's configured ai-proxy-multi instances by model name and optionally provider.
  3. String entries match by model name only. Object entries can match by provider + model for disambiguation (e.g., two providers serving the same model).
  4. Unmatched entries are ignored (the client cannot introduce providers/models not configured on the route).
  5. Configured instances not referenced in models are appended after the client's preferred list, in their original priority order.
  6. The existing fallback_strategy still applies — if the top-priority instance fails, the plugin falls back through the client-specified order, then server-configured instances.
  7. Auth, rate limiting, and all other instance config remain server-controlled.

Configuration

A new optional boolean on the plugin config to opt-in:

plugins:
  ai-proxy-multi:
    allow_client_model_preference: true   # default: false
    instances:
      - name: anthropic-primary
        provider: anthropic
        options:
          model: claude-sonnet-4-20250514
        # ...
      - name: openai-fallback
        provider: openai
        options:
          model: gpt-4o
        # ...

When allow_client_model_preference is false (default), the models field is stripped from the request body and instance ordering is purely server-driven. This ensures backward compatibility.

Use Case

We're building an LLM proxy gateway where multiple teams consume AI services through a shared gateway. Different clients have different model preferences:

  • Team A prefers Claude with GPT-4o fallback
  • Team B prefers GPT-4o with Claude fallback
  • The gateway team manages auth keys, rate limits, and provider configs centrally

Today this requires separate routes per team/preference. With client-driven model selection, a single route handles all teams while respecting their preferences.

Implementation Notes

  • The matching logic would live in ai-proxy-multi.lua, executed before instance selection in the access phase.
  • The models field should be stripped from the request body before forwarding to the upstream provider (providers don't recognize it).
  • String matching should be case-sensitive to match provider model naming conventions.
  • When using chash balancer, client preference should take precedence over hash-based selection.

I'm happy to submit a PR for this feature if the approach looks good.

Lenguaje dominante
Lua
Estrellas
17.3k
Forks
3k
Merge medio
4 d 14 h
PR fusionados (30 d)
73

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/apisix

Todos los issues de apache/apisix

Issues similares

Más issues de Lua

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.