pytorch/text

How to load huge file of data?

Open

#130 opened on Sep 25, 2017

View on GitHub
 (15 comments) (39 reactions) (0 assignees)Python (3,396 stars) (822 forks)batch import
enhancementhelp wanted

Description

Hi guys, is there any plans or tutorial that I can refer to in order to load huge file of data? I have two scenarios in mind that I would probably use

  1. Assuming I have huge files that have already split into several files. How to load all of those files on demand subsequently?
  2. Assuming I have a single huge file. How to load the content on demand?

Thanks guys!

Contributor guide

Tech stack
python
Domain
machine learningdata
Issue type
documentation
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PyTorch basicsData loading concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Review the comments on this issue for any existing solutions or discussions. Check the PyTorch Text documentation and examples for current data loading methods. Consider creating a tutorial or example demonstrating loading large files using chunks or memory mapped files, and update the issue with findings.