Microsoft.ML.TorchSharp QuestionAnswer: training takes the answer's last token as inclusive but the decoder cuts the span as exclusive — every answer comes back without its last token (a one-token answer is the empty string)

オープン
  • 何か問題がありますか？
#7,890 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
72/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
csharp
領域
machine-learning, testing

調査の方向性

Read QATrainer.cs at the training end-index calculation and decoder span slice, then trace the existing QuestionAnswer tests. Add a regression test using a trivially learnable one-token answer set and verify that fitted predictions match the training answers, including the one-token case. The issue is done when decoding returns the complete answer and the regression test passes.

索引モデルが issue の本文から書いたものです。

説明

area-TorchSharp untriaged

Package: Microsoft.ML.TorchSharp 0.23.1 (Microsoft.ML 5.0.x), mlContext.MulticlassClassification.Trainers.QuestionAnswer(...). Related: the NER report #7889 (an off-by-one between what the trainer is taught and what the decoder returns).

Observed. Train on 300 synthetic extractive questions (Where does Leo work? over Leo works at Tyrell in Oslo. It opened last year., answer Tyrell, and Which city is Tyrell in?, answer Oslo), maxEpochs 10; the training loss falls to 0.011. Then answer 60 held-out questions and compare PredictedAnswer[0] with the expected text:

  • exact matches: 0 of 60;
  • empty string: 33; strict prefix of the expected answer (Ty for Tyrell, Umb for Umbrella, Won for Wonka): 27; anything else: 0.

So the span the model finds is right and the decoder cuts its last token.

Cause (read from QATrainer.cs, not stepped through): training computes the inclusive end answerEnd = answerIndex + answer.Length - 1 and the target token of that character; the decoder slices .AsSpan(predictStart - questionLength - 2, predictEnd - predictStart), a length that is correct for an exclusive end. A one-token answer (start == end) therefore decodes to the empty string.

Check that it is the cut and not the model: train the same data with every answer extended through the first character of the next token (a space has no token — the example is skipped — so extend past the whitespace to the next word's first character) and with a . appended to every passage so the last answer also has a next token: the same trainer then answers 56 of 60 held-out questions exactly (exact_match 0.93, char_f1 0.96).

Workaround on our side: that training-side extension plus the passage padding. Models trained with it would answer one token too long once a release decodes the span the way it trains it.

Request: decode the end index as inclusive (predictEnd - predictStart + 1) or train an exclusive end, with a test that a model fitted on a trivially learnable one-token-answer set returns its own training answers.

主要言語
C#
スター
9.4k
フォーク
2k
平均マージ
2日 18時間
マージ済み PR（30日）
64

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dotnet/machinelearning のほかの issue

dotnet/machinelearning の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。