pytorch/vision

[proposal] Move dataset tar/zip extraction and integrity checking of multiple files to utils.py

Open

#441 opened on Mar 6, 2018

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Python (6,858 forks)batch import
enhancementhelp wantedneeds discussion

Repository metrics

Stars
 (15,050 stars)
PR merge metrics
 (Avg merge 12d 8h) (14 merged PRs in 30d)

Description

Datasets such as CIFAR / MNIST / etc have download / integrity logic that is useful for reimplementing custom user datasets (such as https://github.com/vadimkantorov/metriclearningbench/blob/master/cars196.py, https://github.com/vadimkantorov/metriclearningbench/blob/master/cub2011.py, https://github.com/vadimkantorov/metriclearningbench/blob/master/stanford_online_products.py)

I propose moving it to torchvision/datasets/utils.py functions like downloading and extracting tarballs / zipfiles / plain files; checking integrity by md5 of a file list if it is provided.

Currently avoiding duplication leads to quirky subclassing of ImageFolder, Cifar10 etc.

Contributor guide

Research direction
Examine the download and extraction logic in existing datasets like CIFAR and MNIST, then refactor common code into a shared utility function in torchvision/datasets/utils.py.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPython
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70