NVIDIA/DALI

Can dali webdataset reader work out classes from folder structure?

Open

#4,146 opened on Aug 7, 2022

View on GitHub
 (3 comments) (0 reactions) (0 assignees)C++ (670 forks)auto 404
help wanted

Repository metrics

Stars
 (5,722 stars)
PR merge metrics
 (PR metrics pending)

Description

The webdataset reader seems to require a file to define the class per image in a tar? As shown in this example: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/examples/general/data_loading/dataloading_webdataset.html

The standard nvidia.dali.fn.readers.file can work out the class from the folder structure as per: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/operations/nvidia.dali.fn.readers.file.html

Is it possible to get derive the label for a file from the folder that it is in with webdataset or do I need to add an extra file per sample for the class?

Contributor guide

Research direction
Investigate whether the webdataset reader can be extended to support deriving class labels from the folder structure of the shards, similar to the file reader. Examine the existing webdataset reader source code and the file reader's implementation to see what adaptations are needed.
Tech stack
python
Domain
backendmachine learning
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
DALIPythonC++
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50