codertimo/BERT-pytorch

Is it possible to train BERT?

Open

#3 opened on Oct 17, 2018

View on GitHub
 (7 comments) (0 reactions) (1 assignee)Python (5,757 stars) (1,252 forks)batch import
help wantedquestion

Description

Is it possible to achieve the same result as the paper in short time? Well.. I don't have enough GPU & computation power to see the enough result as google ai.

If we can't train the full corpus as the google, then how can we prove that this code is verified? Training 256M size corpus without Google AI class gpu computation is nearly, impossible for me.

If you have any thought(reducing the model size) please let me know!

Contributor guide

Tech stack
pythonpytorch
Domain
machine learning
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Understanding of BERT modelBasic PyTorch knowledge
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate options for training BERT with limited computational resources, such as reducing model size, using smaller batches, or leveraging pre trained weights. Refer to the BERT paper and the repository's training scripts for guidance.