pytorch/ignite

Are there any ways to filter out or ignore a batch in engine?

Open

#996 opened on Apr 30, 2020

View on GitHub
 (7 comments) (1 reaction) (0 assignees)Python (602 forks)batch import
enhancementhelp wanted

Repository metrics

Stars
 (4,313 stars)
PR merge metrics
 (Avg merge 18d 23h) (19 merged PRs in 30d)

Description

❓ Questions/Help/Support

I'm really new to the ML area and I'm trying to train a network with a dataset that when sampling batches, it can create batches with really large sizes (the sampling method is just torch's weightedSampling). Though the batch size is fixed the data size in the batch will be really large sometimes. It seems pretty complicated to define a customized sampler, and the way I originally did was just ignore that batch when the data size in the batch is too large. Now, I'm trying to use ignite as the framework to train the network, I don't know how to do this. I think it might relate to event_filter? But it seems to be triggered by event, so are there any ways to pass in a batch instead?

Thanks so much for your time!

Contributor guide

Research direction
Investigate Ignite's event filtering mechanism and consider customizing the DataLoader to skip large batches. Look at examples of using `event filter` or overriding the batch provider.
Tech stack
pythonpytorch
Domain
machine learningai
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonPyTorchIgniteGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70