Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

/v1/tokenize returns a flat list when the text array holds a single string

Abierto
  • ¿Algo no está bien?
#4,664 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@Daksha1611 ya está trabajando en esto.

Desde el 17/10/2026.

  • #4665 de @Daksha1611 — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
cpp, json
Área
backend-api-design, documentation, testing-qa

Línea de trabajo

Start with src/tokenize/tokenize_parser.cpp, especially the nesting condition based on outputShape[0] > 1, and compare it with the request parsing sequence in src/llm/servable.cpp. Update or add coverage in tokenize_parser_test.cpp and tokenize_endpoint_test.cpp for a one-element array, then update the tokenize response documentation. Done means every string array returns one token list per input, including a single string, with the endpoint behavior and documented response shape covered.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

The shape of the /v1/tokenize response follows the number of strings in the request instead of the
form of the request. src/tokenize/tokenize_parser.cpp nests the tokens only when the tokenizer
output has more than one row (outputShape[0] > 1), and a one-element array produces one row, the
same as a plain string:

{"text": "hello world"}              ->  {"tokens":[14990,1879]}
{"text": ["hello world"]}           ->  {"tokens":[14990,1879]}
{"text": ["hello world", "hi"]}     ->  {"tokens":[[14990,1879],[6023]]}

A client that sends a list and reads tokens[i] for each input gets a list of token lists for
two or more inputs, but a flat list of integers for exactly one. Batches that happen to contain one
string need special handling on the client.

None of the tests send a one-element array: the list cases in tokenize_parser_test.cpp and
tokenize_endpoint_test.cpp all use two or more strings. The docs show only the single-string
response.

To Reproduce

curl http://localhost:8000/v1/tokenize -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "text": ["hello world"]}'

Reproduced with the parser and a real Qwen3 tokenizer, following the same sequence as
src/llm/servable.cpp: parse the request, encode, build the response.

Expected behavior

An array of strings always gives a list of token lists, including an array with one string.

Note that this changes the response for clients that currently send a one-element array and rely on
the flat list.

Logs

None.

Configuration

  1. OVMS version: main @ eec7daac
  2. any servable that serves /v1/tokenize (LLM, VLM, embeddings, rerank)
  3. N/A
  4. N/A
  5. N/A

Additional context

I have a patch with tests and docs and will open a PR.

Lenguaje dominante
C++
Estrellas
952
Forks
285
Merge medio
3 d 5 h
PR fusionados (30 d)
45

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de openvinotoolkit/model_server

Todos los issues de openvinotoolkit/model_server

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.