Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Responses API (`/v1/responses`) and Vertex AI: model access for a multi-turn benchmark

Đang mở
  • Có gì không ổn?
#1,033 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
15/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
api

Hướng nghiên cứu

This is a design proposal, so start by deciding the approach rather than writing code. Read models/clients/factory.py (the _SUPPORTED_PROVIDER_TYPES check), models/clients/adapters/openai_compatible.py (where _ROUTE_CHAT is fixed to /chat/completions) and models/clients/parsing.py (where parse_assistant_message drops reasoning items). Done means a maintainer-agreed design covering routing, passed-through fields and replay of reasoning items.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Priority Level: Medium (Nice to have)

Is your feature request related to a problem? Please describe.

OpenAI models already work through the OpenAI-compatible adapter (Chat Completions). This is only about
OpenAI's other endpoint, /v1/responses, and only for multi-turn calls to reasoning models.

The benchmark below needs two things from Data Designer: the Responses API (this section), and Vertex AI
(see "Second gap: Vertex AI" below, folded in from #92).

Some evaluations send a model its own earlier reasoning back on the next turn, because models trained on
preserved thinking expect it. MindEval (Apache-2.0), a
multi-session counselling benchmark that runs on NeMo User Sim, does this for the model under test, and
User Sim now offers the same as an opt-in.

OpenAI's reasoning models return their reasoning only through the Responses API, as reasoning items
(encrypted when the call asks for include: ["reasoning.encrypted_content"] with store: false). Through
Chat Completions they return none. So through Data Designer, which reaches OpenAI-type endpoints with its
OpenAI-compatible adapter, these models can be called but their reasoning cannot be carried to the next
turn: they are evaluated without the context they would have in deployment, and their scores are not
comparable with runs that carry it.

Today MindEval works around this by calling those models through litellm's Responses bridge
(openai/responses/<model>, azure/responses/<deployment>) instead of Data Designer. Its README:

OpenAI reasoning models return it only through the Responses API: run them as
azure/responses/<deployment> (or openai/responses/<model>) with
--counselor_params '{"include": ["reasoning.encrypted_content"], "store": false}'.

Describe the solution you'd like

A way for a Data Designer model to use the Responses API and hand back what is needed to continue the
conversation, for example:

  • a provider or endpoint option that sends a model's calls to /v1/responses (OpenAI and Azure OpenAI)
  • request fields passed through (include, store, reasoning)
  • the reasoning items returned on the response, and accepted again on the earlier assistant turns of the
    next request, so a caller can replay them

If there is already a way to do this in Data Designer, a pointer would be just as good, which is why
this is filed as a question as much as a request.

Describe alternatives you've considered
  • Chat Completions only: works, but the reasoning is lost between turns (the problem above).
  • litellm's Responses bridge outside Data Designer: what MindEval does now. It works, but those calls
    bypass Data Designer's model layer, so User Sim's accounting, retries and model configuration don't
    apply to them.
  • A proxy in front of Data Designer (e.g. a LiteLLM proxy translating /chat/completions to
    /responses): gets the call through, but the reasoning items come back as an extra field that Data
    Designer's parser drops, and there is no way to send them back on the next turn.
  • previous_response_id with store: true: keeps the reasoning on OpenAI's side instead of
    carrying it. Not usable where data must not be stored, and it ties each turn to one provider session.
Agent Investigation

Checked on main (2026-10-14) and in 0.9.4:

  • models/clients/factory.py: _SUPPORTED_PROVIDER_TYPES = ("openai", "anthropic"); any other
    provider type raises. Plugins can't add a model client (plugin types: column generator, seed reader,
    processor).
  • models/clients/adapters/openai_compatible.py: chat calls always post to _ROUTE_CHAT = "/chat/completions"; no other route for chat.
  • models/clients/parsing.py: parse_assistant_message keeps content, reasoning_content (text),
    tool_calls and images; other message fields, such as reasoning items, are dropped.
  • No existing issue found ("responses api", "encrypted_content", "/v1/responses").
  • In MindEval (mindeval/llm.py), litellm returns the items as reasoning_items (or under
    provider_specific_fields) on the message; the benchmark stores them with the turn and sends them back
    on that turn next time (mindeval/session.py, CARRIED = ("reasoning_content", "thinking_blocks", "reasoning_items")).
Second gap: Vertex AI

The same benchmark uses Claude and Gemini on Vertex AI, today through litellm's vertex_ai/ provider,
outside Data Designer's model layer. What blocks it, as far as we can see in the code (main):

  • Claude on Vertex (per Google Cloud's API docs) speaks the Anthropic Messages format, but at
    …/publishers/anthropic/models/{model}:rawPredict, with the model in the URL,
    anthropic_version: "vertex-2023-10-16" in the body, and a Google access token. The Anthropic adapter
    (models/clients/adapters/anthropic.py) posts to {endpoint}/v1/messages with x-api-key and the
    model in the body.
  • Gemini on Vertex works through Vertex's OpenAI-compatible endpoint, but authenticates with a Google
    access token that expires after about an hour, while a provider's key is resolved once when the client
    is built (models/clients/factory.py, _resolve_api_key). A long run fails once the token expires.

Useful pieces: Google credentials (Application Default Credentials or a service account) with token
refresh during a run, and the Vertex URL and body shape for Claude.

Additional context

Related: User Sim's opt-in replay of the assistant's own reasoning (NVIDIA-NeMo/UserSim#31) currently
covers OpenAI-compatible endpoints that return reasoning_content; this would extend it to OpenAI's own
reasoning models.

  • I've reviewed existing issues and the documentation
  • This is a design proposal, not a "please build this" request
Ngôn ngữ chính
Python
Star
2.4k
Fork
234
Merge trung bình
23 giờ
Pull request đã merge (30 ngày)
75

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA-NeMo/DataDesigner

Tất cả issue của NVIDIA-NeMo/DataDesigner

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.