pytorch/vision

[RFC] Batteries Included - Phase 3

Open

#6,323 opened on Jul 27, 2022

View on GitHub
 (52 comments) (12 reactions) (0 assignees)Python (15,050 stars) (6,858 forks)batch import
help wantedmodule: modelsmodule: opsmodule: reference scriptsmodule: transformsnew feature

Description

🚀 The feature

Note: To track the progress of the project check out this board.

This is the 3rd phase of TorchVision's modernization project (see phase 1 and 2). We aim to keep TorchVision relevant by ensuring it provides off-the-shelf all the necessary primitives, model architectures and recipe utilities to produce SOTA results for the supported Computer Vision tasks.

To enable our users to reproduce the latest state-of-the-art research we will enhance TorchVision with the following data augmentations, layers, losses and other operators:

Data Augmentations

Losses

Operators added in PyTorch Core

To ensure that our users have access to the most popular SOTA models, we will add the following architectures along with pre-trained weights:

Image Classification

Video Classification

  • MViTv2 [1] - #6373
  • Swin3d [1] - #6499 #6521
  • S3D [1] - #6402 #6412 #6537

To ensure that are users can have access to strong baselines and SOTA weights, we will improve our training recipes to incorporate the newly released primitives and offer improved pre-trained models:

Reference Scripts

  • Update the Reference Scripts to use the latest primitives - #6405 #6433

Pre-trained weights

  • Improve the accuracy of Video models

Other Candidates

There are several other Operators (#5414), Losses (#2980), Augmentations (#3817) and Models (#2707) proposed by the community. Here are some potential candidates that we could implement depending on bandwidth. Contributions are welcome for any of the below:

cc @datumbox @vfdev-5

Contributor guide

Tech stack
python
Domain
machine learningai
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
unclear
Prerequisites
PyTorch knowledgeComputer vision basicsFamiliarity with data augmentationsModel architecture understanding
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
This is a meta issue tracking Phase 3 of the TorchVision modernization. It lists many sub tasks such as new primitives (auto augment, mosaic, dice loss, etc.), new model architectures (MobileViT, etc.), and improved training recipes. Each sub task has linked issues and PRs. To contribute, a newcomer should first identify a specific sub task from the checkboxes, read the linked issues, and understand the scope. The issue is not self contained; it requires investigation into specific sub items.