pytorch/text

How to load huge file of data?

Open

#130 建立於 2017年9月25日

在 GitHub 查看
 (15 留言) (39 反應) (0 負責人)Python (3,396 star) (822 fork)batch import
enhancementhelp wanted

描述

Hi guys, is there any plans or tutorial that I can refer to in order to load huge file of data? I have two scenarios in mind that I would probably use

  1. Assuming I have huge files that have already split into several files. How to load all of those files on demand subsequently?
  2. Assuming I have a single huge file. How to load the content on demand?

Thanks guys!

貢獻者指南

技術棧
python
領域
machine learningdata
議題類型
documentation
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
2
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-3 hours
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
PyTorch basicsData loading concepts
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
70
研究方向
Review the comments on this issue for any existing solutions or discussions. Check the PyTorch Text documentation and examples for current data loading methods. Consider creating a tutorial or example demonstrating loading large files using chunks or memory mapped files, and update the issue with findings.
How to load huge file of data? · pytorch/text#130 | Good First Issue