Microsoft.ML.TorchSharp QuestionAnswer: training takes the answer's last token as inclusive but the decoder cuts the span as exclusive — every answer comes back without its last token (a one-token answer is the empty string)

Aperta
  • Qualcosa non va?
#7,890 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
72/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
machine-learning, testing

Direzione di ricerca

Read QATrainer.cs at the training end-index calculation and decoder span slice, then trace the existing QuestionAnswer tests. Add a regression test using a trivially learnable one-token answer set and verify that fitted predictions match the training answers, including the one-token case. The issue is done when decoding returns the complete answer and the regression test passes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area-TorchSharp untriaged

Package: Microsoft.ML.TorchSharp 0.23.1 (Microsoft.ML 5.0.x), mlContext.MulticlassClassification.Trainers.QuestionAnswer(...). Related: the NER report #7889 (an off-by-one between what the trainer is taught and what the decoder returns).

Observed. Train on 300 synthetic extractive questions (Where does Leo work? over Leo works at Tyrell in Oslo. It opened last year., answer Tyrell, and Which city is Tyrell in?, answer Oslo), maxEpochs 10; the training loss falls to 0.011. Then answer 60 held-out questions and compare PredictedAnswer[0] with the expected text:

  • exact matches: 0 of 60;
  • empty string: 33; strict prefix of the expected answer (Ty for Tyrell, Umb for Umbrella, Won for Wonka): 27; anything else: 0.

So the span the model finds is right and the decoder cuts its last token.

Cause (read from QATrainer.cs, not stepped through): training computes the inclusive end answerEnd = answerIndex + answer.Length - 1 and the target token of that character; the decoder slices .AsSpan(predictStart - questionLength - 2, predictEnd - predictStart), a length that is correct for an exclusive end. A one-token answer (start == end) therefore decodes to the empty string.

Check that it is the cut and not the model: train the same data with every answer extended through the first character of the next token (a space has no token — the example is skipped — so extend past the whitespace to the next word's first character) and with a . appended to every passage so the last answer also has a next token: the same trainer then answers 56 of 60 held-out questions exactly (exact_match 0.93, char_f1 0.96).

Workaround on our side: that training-side extension plus the passage padding. Models trained with it would answer one token too long once a release decodes the span the way it trains it.

Request: decode the end index as inclusive (predictEnd - predictStart + 1) or train an exclusive end, with a test that a model fitted on a trivially learnable one-token-answer set returns its own training answers.

Lingua principale
C#
Stelle
9.4k
Fork
2k
Merge medio
2g 18h
PR unite (30g)
64

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di dotnet/machinelearning

Tutte le issue di dotnet/machinelearning

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.