pytorch/text

Language modelling dataset only one sample

Open

#381 opened on Sep 12, 2018

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Python (3,396 stars) (822 forks)batch import
datasetshelp wanted

Description

from torchtext import data
from torchtext import datasets

TEXT = data.Field(lower=True, batch_first=True)

train, valid, test = datasets.WikiText2.splits(TEXT)

print('len(train)', len(train))

This returns a length of one. It should print the length of the whole dataset. I have tried both with version 0.2.3 and 0.3 and none of them worked.

Contributor guide

Tech stack
pythonpytorch
Domain
data
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Python programmingPyTorch basicstorchtext dataset API
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50
Research direction
Investigate the implementation of the `splits` method or the WikiText2 dataset class in the torchtext library. The bug likely resides in how the dataset is loaded or split, causing only one sample to be returned. Examine the source code in the `torchtext/datasets/` directory and check for any version specific issues between 0.2.3 and 0.3. Also review any related issues or documentation to understand expected behavior.