Microsoft.ML.TorchSharp QuestionAnswer: training takes the answer's last token as inclusive but the decoder cuts the span as exclusive — every answer comes back without its last token (a one-token answer is the empty string)

Aberta
  • Algo errado?
#7,890 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
72/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
csharp
Domínio
machine-learning, testing

Direção de pesquisa

Read QATrainer.cs at the training end-index calculation and decoder span slice, then trace the existing QuestionAnswer tests. Add a regression test using a trivially learnable one-token answer set and verify that fitted predictions match the training answers, including the one-token case. The issue is done when decoding returns the complete answer and the regression test passes.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

area-TorchSharp untriaged

Package: Microsoft.ML.TorchSharp 0.23.1 (Microsoft.ML 5.0.x), mlContext.MulticlassClassification.Trainers.QuestionAnswer(...). Related: the NER report #7889 (an off-by-one between what the trainer is taught and what the decoder returns).

Observed. Train on 300 synthetic extractive questions (Where does Leo work? over Leo works at Tyrell in Oslo. It opened last year., answer Tyrell, and Which city is Tyrell in?, answer Oslo), maxEpochs 10; the training loss falls to 0.011. Then answer 60 held-out questions and compare PredictedAnswer[0] with the expected text:

  • exact matches: 0 of 60;
  • empty string: 33; strict prefix of the expected answer (Ty for Tyrell, Umb for Umbrella, Won for Wonka): 27; anything else: 0.

So the span the model finds is right and the decoder cuts its last token.

Cause (read from QATrainer.cs, not stepped through): training computes the inclusive end answerEnd = answerIndex + answer.Length - 1 and the target token of that character; the decoder slices .AsSpan(predictStart - questionLength - 2, predictEnd - predictStart), a length that is correct for an exclusive end. A one-token answer (start == end) therefore decodes to the empty string.

Check that it is the cut and not the model: train the same data with every answer extended through the first character of the next token (a space has no token — the example is skipped — so extend past the whitespace to the next word's first character) and with a . appended to every passage so the last answer also has a next token: the same trainer then answers 56 of 60 held-out questions exactly (exact_match 0.93, char_f1 0.96).

Workaround on our side: that training-side extension plus the passage padding. Models trained with it would answer one token too long once a release decodes the span the way it trains it.

Request: decode the end index as inclusive (predictEnd - predictStart + 1) or train an exclusive end, with a test that a model fitted on a trivially learnable one-token-answer set returns its own training answers.

Linguagem predominante
C#
Estrelas
9.4k
Forks
2k
Merge médio
2d 18h
PRs com merge (30d)
64

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de dotnet/machinelearning

Todas as issues de dotnet/machinelearning

Issues semelhantes

Mais issues de C#

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.