pytorch/examples
Inconsistency in Evaluation of Word Language Model
Ouverte
#214 ouverte le 8 sept. 2017
good first issue
Métriques du dépôt
- Stars
- (21 634 étoiles)
- Métriques de merge PR
- (Métriques PR en attente)
Description
I was looking at the main.py code for word-level language modeling and noticed a possible inconsistency. The final evaluation loss is intended to be a mean of the individual losses implemented as a weighted mean of the batches with the weight being the sequence length of the batch.
There are len(data_source)-1 such losses.
In the end, however, the division is performed with len(data_source) causing an inconsistency.
A similar issue also arises with the book-keeping in the training loss. If this is true, the fix should be straightforward, we would need to keep track of total_seen and divide by that instead of some pre-determined quantity in both training and evaluation cases.
Tagging: @Smerity