codertimo/BERT-pytorch

chooses 15% of token

Open

#56 opened on Feb 22, 2019

View on GitHub
 (1 comment) (1 reaction) (0 assignees)Python (5,757 stars) (1,252 forks)batch import
good first issue

Description

From paper, it mentioned

Instead, the training data generator chooses 15% of tokens at random, e.g., in the sentence my dog is hairy it chooses hairy.

It means that 15% of token will be choose for sure.

From https://github.com/codertimo/BERT-pytorch/blob/master/bert_pytorch/dataset/dataset.py#L68, for every single token, it has 15% of chance that go though the followup procedure. Does it aligned with 15% of token will be chosen?

Contributor guide

Tech stack
pythonpytorch
Domain
machine learningai
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
BERT masking knowledgePythonPyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Examine the code at bert pytorch/dataset/dataset.py line 68 and compare with the BERT paper's description of selecting 15% of tokens randomly. Determine if the current implementation selects exactly 15% of tokens per sequence or if it uses a per token probability that results in an average of 15% but may not guarantee exactly 15% per sequence. Consider whether the implementation aligns with the paper's intention and if a fix is needed to ensure deterministic 15% selection.