pytorch/examples

Inconsistency in Evaluation of Word Language Model

Open

#214 opened on Sep 8, 2017

View on GitHub
 (3 comments) (3 reactions) (0 assignees)Python (9,429 forks)batch import
good first issue

Repository metrics

Stars
 (21,634 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

I was looking at the main.py code for word-level language modeling and noticed a possible inconsistency. The final evaluation loss is intended to be a mean of the individual losses implemented as a weighted mean of the batches with the weight being the sequence length of the batch.

https://github.com/pytorch/examples/blob/930ae27d64ceae1c77bbf616e713bc4b7c403849/word_language_model/main.py#L116

There are len(data_source)-1 such losses.

https://github.com/pytorch/examples/blob/930ae27d64ceae1c77bbf616e713bc4b7c403849/word_language_model/main.py#L112

In the end, however, the division is performed with len(data_source) causing an inconsistency.

https://github.com/pytorch/examples/blob/930ae27d64ceae1c77bbf616e713bc4b7c403849/word_language_model/main.py#L118

A similar issue also arises with the book-keeping in the training loss. If this is true, the fix should be straightforward, we would need to keep track of total_seen and divide by that instead of some pre-determined quantity in both training and evaluation cases.

Tagging: @Smerity

Contributor guide

Research direction
Examine the evaluation code in word language model/main.py to confirm the inconsistency in loss calculation and implement the fix by tracking total seen instead of dividing by len(data source).
Tech stack
python
Domain
machine learning
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonPyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75