pytorch/vision

[proposal] Move dataset tar/zip extraction and integrity checking of multiple files to utils.py

Open

#441 opened on Mar 6, 2018

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Python (15,050 stars) (6,858 forks)batch import
enhancementhelp wantedneeds discussion

Description

Datasets such as CIFAR / MNIST / etc have download / integrity logic that is useful for reimplementing custom user datasets (such as https://github.com/vadimkantorov/metriclearningbench/blob/master/cars196.py, https://github.com/vadimkantorov/metriclearningbench/blob/master/cub2011.py, https://github.com/vadimkantorov/metriclearningbench/blob/master/stanford_online_products.py)

I propose moving it to torchvision/datasets/utils.py functions like downloading and extracting tarballs / zipfiles / plain files; checking integrity by md5 of a file list if it is provided.

Currently avoiding duplication leads to quirky subclassing of ImageFolder, Cifar10 etc.

Contributor guide

Tech stack
python
Domain
data
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
familiarity with torchvision datasets module
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Review the existing dataset implementations in torchvision/datasets, such as CIFAR and MNIST, to identify common download and extraction patterns. Then refactor those patterns into utility functions in a new or existing utils.py file within the same directory. Ensure that the new functions accept parameters for URLs, file paths, and integrity checks (e.g., MD5). The issue (pytorch/vision#441) provides motivation and links to example custom datasets that currently duplicate this logic.