pytorch/examples

Is the loss of the first word covered during the language model evaluation?

Open

#453 opened on Nov 26, 2018

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (9,429 forks)batch import
good first issuenlp

Repository metrics

Stars
 (21,634 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

In the language model example, it seems that during the evaluation, the code starts from computing the loss of the second word. Thus, skipping the loss of the first word. https://github.com/pytorch/examples/blob/537f6971872b839b36983ff40dafe688276fe6c3/word_language_model/main.py#L136 https://github.com/pytorch/examples/blob/537f6971872b839b36983ff40dafe688276fe6c3/word_language_model/main.py#L121-L125

Furthermore, the evaluation data is divided into 10 batches, hence, the losses of 10 words are skipped. Am I right or I did miss something? https://github.com/pytorch/examples/blob/537f6971872b839b36983ff40dafe688276fe6c3/word_language_model/main.py#L85-L88

Contributor guide

Research direction
Examine the evaluation loop in main.py lines 85-88 and 136, and compare with the training loop to verify if the first token's loss is omitted. Trace the data flow of inputs and targets to see if the model predicts the next word from the first word but computes loss starting from the second word.
Tech stack
pythonpytorch
Domain
machine learning
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonPyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60