Prompt-token deduplication drops distinct image-conditioned transitions

Abierto
  • ¿Algo no está bien?
#695 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

@liuzhengyang699 ya está trabajando en esto.

Desde el 6/11/2026.

  • #696 de @liuzhengyang699 — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
1-2 días
Aptitud para principiantes
20/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
python
Área
machine-learning

Línea de trabajo

Start by tracing the triplet event view's deduplication key and the rollout manager's image URL alignment, then inspect the regression tests described in the issue. Reproduce with the Qwen2-VL processor and tokenizer on CPU, using same-image retries and distinct or reordered images. Done means separate images retain their triplets and routes through the API-to-training-batch path, while same-image retries keep the latest request and trimmed output omits image data URLs.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

The triplet event view deduplicates model requests using only prompt token IDs. Image placeholder IDs do not encode the image pixels, so two different images can have the same prompt IDs and the earlier transition is silently discarded. The rollout manager mirrors that token-only key when aligning image URLs.

I reproduced this on bc3a9e8 using a red and a blue 256×256 image, the Qwen2-VL image processor, and its tokenizer/placeholder expansion rule. The prompts have the same 88 token IDs, while the pixel tensors differ. Posting requests for red, a red retry, then blue through the real event API gives:

  • Actual: three raw requests → one blue triplet → one training row.
  • Expected: two triplets/rows, keeping the latest red request and the separate blue request.

The same image distinction is needed when compacting routed-expert payloads. Keeping both image requests while deleting the first one's routes causes require_routed_experts=True to fail during batch construction.

A focused fix uses prompt IDs plus ordered image URLs for event deduplication and manager alignment, and retains routes when image inputs differ. The trimmed event schema stays unchanged, so image data URLs do not appear in triplet output. Regression tests cover same-image retries, different/reordered images, serialized message content, and the real API → manager → image decoding → training-batch path with and without required routes.

This was found during a follow-up audit with AI assistance. The processor reproduction and tests run on CPU; no model weights, GPU inference, or end-to-end training were used.

Lenguaje dominante
Python
Estrellas
18.9k
Forks
1.7k
Merge medio
5 d 19 h
PR fusionados (30 d)
14

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/agent-lightning

Todos los issues de microsoft/agent-lightning

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.