codertimo/BERT-pytorch

pred_loss decrease fast while avg_acc stay at 50%

Open

#32 opened on Oct 23, 2018

View on GitHub
 (53 comments) (1 reaction) (0 assignees)Python (5,757 stars) (1,252 forks)batch import
help wantedinvalidquestion

Description

I try to run the code on a small dataset and I find that pred_loss decrease fast while avg_acc stay at 50%. It is strange to me since decrease in pred_loss should indicates increase in accuracy. image

Contributor guide

Tech stack
pythonpytorch
Domain
machine learningai
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
basic understanding of BERTPyTorch experience
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The issue reports that pred loss decreases but accuracy stays at 50%. First, check the training loop in the codebase (likely in a file like 'trainer.py' or similar) to understand how loss and accuracy are computed. Look for potential mismatches: e.g., loss might be computed on a different sub task (next sentence prediction vs masked LM) or accuracy might be averaged over batches without proper weighting. Also, examine the dataset; if it's highly imbalanced, accuracy can remain near 50% while loss decreases due to the model learning only the majority class. Review the comments in the issue for any debugging steps already taken by the reporter or others. Finally, consider if the problem is due to a bug in the accuracy calculation (e.g., thresholding or argmax applied incorrectly).