Prompt-token deduplication drops distinct image-conditioned transitions

Aperta
  • Qualcosa non va?
#695 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@liuzhengyang699 ci sta già lavorando.

Dal 6/11/2026.

  • #696 di @liuzhengyang699 — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
20/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
python
Ambito
machine-learning

Direzione di ricerca

Start by tracing the triplet event view's deduplication key and the rollout manager's image URL alignment, then inspect the regression tests described in the issue. Reproduce with the Qwen2-VL processor and tokenizer on CPU, using same-image retries and distinct or reordered images. Done means separate images retain their triplets and routes through the API-to-training-batch path, while same-image retries keep the latest request and trimmed output omits image data URLs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

The triplet event view deduplicates model requests using only prompt token IDs. Image placeholder IDs do not encode the image pixels, so two different images can have the same prompt IDs and the earlier transition is silently discarded. The rollout manager mirrors that token-only key when aligning image URLs.

I reproduced this on bc3a9e8 using a red and a blue 256×256 image, the Qwen2-VL image processor, and its tokenizer/placeholder expansion rule. The prompts have the same 88 token IDs, while the pixel tensors differ. Posting requests for red, a red retry, then blue through the real event API gives:

  • Actual: three raw requests → one blue triplet → one training row.
  • Expected: two triplets/rows, keeping the latest red request and the separate blue request.

The same image distinction is needed when compacting routed-expert payloads. Keeping both image requests while deleting the first one's routes causes require_routed_experts=True to fail during batch construction.

A focused fix uses prompt IDs plus ordered image URLs for event deduplication and manager alignment, and retains routes when image inputs differ. The trimmed event schema stays unchanged, so image data URLs do not appear in triplet output. Regression tests cover same-image retries, different/reordered images, serialized message content, and the real API → manager → image decoding → training-batch path with and without required routes.

This was found during a follow-up audit with AI assistance. The processor reproduction and tests run on CPU; no model weights, GPU inference, or end-to-end training were used.

Lingua principale
Python
Stelle
18.9k
Fork
1.7k
Merge medio
5g 19h
PR unite (30g)
14

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/agent-lightning

Tutte le issue di microsoft/agent-lightning

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.