huggingface/accelerate

Barebones dataloader to allow for any type of iterable dataloader-like object to be used. Should just handle device placement

Open

#2,975 opened on Jul 29, 2024

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Python (5,805 stars) (626 forks)batch import
good first issue

Description

Make a simplistic version of the DispatchDataLoader which allows for users to easily pass in any Iterable type of object and call it's __iter__. Likely situation to make sure Accelerator knows what's up during prepare is to add an option to the DataLoaderConfiguration for custom_classes which accepts types for us to do isinstance() on

Contributor guide

Tech stack
pythonpytorch
Domain
machine learning
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Familiarity with PyTorch DataLoaderUnderstanding of accelerate's DataLoaderConfiguration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Review existing DispatchDataLoader implementation in accelerate/src/accelerate/data loader.py. Understand the DataLoaderConfiguration class in the same file. Check comments for desired interface. Implement a simple iterable wrapper that handles device placement, and add custom classes option to DataLoaderConfiguration.
Barebones dataloader to allow for any type of iterable dataloader-like object to be used. Should just handle device placement · huggingface/accelerate#2975 | Good First Issue