Microsoft.ML.TorchSharp QuestionAnswer: training takes the answer's last token as inclusive but the decoder cuts the span as exclusive — every answer comes back without its last token (a one-token answer is the empty string)

未关闭
  • 哪里不对？
#7,890 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
72/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
csharp
领域
machine-learning, testing

调研方向

Read QATrainer.cs at the training end-index calculation and decoder span slice, then trace the existing QuestionAnswer tests. Add a regression test using a trivially learnable one-token answer set and verify that fitted predictions match the training answers, including the one-token case. The issue is done when decoding returns the complete answer and the regression test passes.

由索引模型根据 Issue 内容生成。

描述

area-TorchSharp untriaged

Package: Microsoft.ML.TorchSharp 0.23.1 (Microsoft.ML 5.0.x), mlContext.MulticlassClassification.Trainers.QuestionAnswer(...). Related: the NER report #7889 (an off-by-one between what the trainer is taught and what the decoder returns).

Observed. Train on 300 synthetic extractive questions (Where does Leo work? over Leo works at Tyrell in Oslo. It opened last year., answer Tyrell, and Which city is Tyrell in?, answer Oslo), maxEpochs 10; the training loss falls to 0.011. Then answer 60 held-out questions and compare PredictedAnswer[0] with the expected text:

  • exact matches: 0 of 60;
  • empty string: 33; strict prefix of the expected answer (Ty for Tyrell, Umb for Umbrella, Won for Wonka): 27; anything else: 0.

So the span the model finds is right and the decoder cuts its last token.

Cause (read from QATrainer.cs, not stepped through): training computes the inclusive end answerEnd = answerIndex + answer.Length - 1 and the target token of that character; the decoder slices .AsSpan(predictStart - questionLength - 2, predictEnd - predictStart), a length that is correct for an exclusive end. A one-token answer (start == end) therefore decodes to the empty string.

Check that it is the cut and not the model: train the same data with every answer extended through the first character of the next token (a space has no token — the example is skipped — so extend past the whitespace to the next word's first character) and with a . appended to every passage so the last answer also has a next token: the same trainer then answers 56 of 60 held-out questions exactly (exact_match 0.93, char_f1 0.96).

Workaround on our side: that training-side extension plus the passage padding. Models trained with it would answer one token too long once a release decodes the span the way it trains it.

Request: decode the end index as inclusive (predictEnd - predictStart + 1) or train an exclusive end, with a test that a model fitted on a trivially learnable one-token-answer set returns its own training answers.

主要语言
C#
星标
9.4k
派生
2k
平均合并
2 天 18 小时
30 天内合并 PR
64

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

dotnet/machinelearning 的其他 Issue

查看 dotnet/machinelearning 的全部 Issue

相似的 Issue

更多 C# Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。